Ameet Talwalkar

Ameet Talwalkar

Auto-generated profile, based on publicly available information

Ameet Talwalkar

Carnegie Mellon University, Pittsburgh, Pennsylvania, USA

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications

19 April 2023
Verified
Conference Paper
Open access
ÁCÁngel Alexander Cabrera
Ángel Alexander Cabrera
EFErica Fu
Erica Fu
DBDonald Bertucci
Donald Bertucci
KHKenneth Holstein
Kenneth Holstein
ATAmeet Talwalkar
Ameet Talwalkar
APAdam Perer
Adam Perer
Machine learning models with high accuracy on test data can still produce systematic failures, such as harmful biases and safety issues, when deployed in the real world. To detect and mitigate such failures, practitioners run behavioral evaluation of their models, checking model outputs for specific types of inputs. Behavioral evaluation is important but challenging, requiring that practitioners discover real-world patterns and validate systematic failures. We conducted 18 semi-structured interviews with ML practitioners to better understand the challenges of behavioral evaluation and found that it is a collaborative, use-case-first process that is not adequately supported by existing task- and domain-specific tools. Using these findings, we designed zeno, a general-purpose framework for visualizing and testing AI systems across diverse use cases. In four case studies with participants using zeno on real-world models, we found that practitioners were able to reproduce previous manual analyses and discover new systematic failures.

25 April 2025
Verified
Conference Paper
Open access
VCValerie Chen
Valerie Chen
AZAlan Zhu
Alan Zhu
SZSebastian Zhao
Sebastian Zhao
HMHussein Mozannar
Hussein Mozannar
DSDavid Sontag
David Sontag
ATAmeet Talwalkar
Ameet Talwalkar

19 June 2018 in arXiv
Verified
Preprint
Open access
PDF file
LLLisha Li
Lisha Li
KJKevin Jamieson
Kevin Jamieson
GDGiulia DeSalvo
Giulia DeSalvo
ARAfshin Rostamizadeh
Afshin Rostamizadeh
ATAmeet Talwalkar
Ameet Talwalkar
Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter optimization as a pure-exploration non-stochastic infinite-armed bandit problem where a predefined resource like iterations, data samples, or features is allocated to randomly sampled configurations. We introduce a novel algorithm, Hyperband, for this framework and analyze its theoretical properties, providing several desirable guarantees. Furthermore, we compare Hyperband with popular Bayesian optimization methods on a suite of hyperparameter optimization problems. We observe that Hyperband can provide over an order-of-magnitude speedup over our competitor set on a variety of deep-learning and kernel-based learning problems.

24 March 2024 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
KAKasun Amarasinghe
Kasun Amarasinghe
KRKit T. Rodolfa
Kit T. Rodolfa
SJSérgio Jesus
Sérgio Jesus
VCValerie Chen
Valerie Chen
VBVladimir Balayan
Vladimir Balayan
RGRayid Ghani
Rayid Ghani
Most existing evaluations of explainable machine learning (ML) methods rely on simplifying assumptions or proxies that do not reflect real-world use cases; the handful of more robust evaluations on real-world settings have shortcomings in their design, generally leading to overestimation of methods' real-world utility. In this work, we seek to address this by conducting a study that evaluates post-hoc explainable ML methods in a setting consistent with the application context and provide a template for future evaluation studies. We modify and improve a prior study on e-commerce fraud detection by relaxing the original work's simplifying assumptions that departed from the deployment context. Our study finds no evidence for the utility of the tested explainable ML methods in the context, which is a drastically different conclusion from the earlier work. This highlights how seemingly trivial experimental design choices can yield misleading conclusions about method utility. In addition, our work carries lessons about the necessity of not only evaluating explainable ML methods using tasks, data, users, and metrics grounded in the intended application context but also developing methods tailored to specific applications, moving beyond general-purpose explainable ML methods.

3 November 2023 in Proceedings of the AAAI Conference on Human Computation and Crowdsourcing
Verified
Journal Article
Open access
PDF file
NJNari Johnson
Nari Johnson
ÁCÁngel Alexander Cabrera
Ángel Alexander Cabrera
GPGregory Plumb
Gregory Plumb
ATAmeet Talwalkar
Ameet Talwalkar
Machine learning (ML) models that achieve high average accuracy can still underperform on semantically coherent subsets ("slices") of data. This behavior can have significant societal consequences for the safety or bias of the model in deployment, but identifying these underperforming slices can be difficult in practice, especially in domains where practitioners lack access to group annotations to define coherent subsets of their data. Motivated by these challenges, ML researchers have developed new slice discovery algorithms that aim to group together coherent and high-error subsets of data. However, there has been little evaluation focused on whether these tools help humans form correct hypotheses about where (for which groups) their model underperforms. We conduct a controlled user study (N = 15) where we show 40 slices output by two state-of-the-art slice discovery algorithms to users, and ask them to form hypotheses about an object detection model. Our results provide positive evidence that these tools provide some benefit over a naive baseline, and also shed light on challenges faced by users during the hypothesis formation step. We conclude by discussing design opportunities for ML and HCI researchers. Our findings point to the importance of centering users when creating and evaluating new tools for slice discovery.