Erica Fu

Erica Fu

Auto-generated profile, based on publicly available information

Erica Fu

Carnegie Mellon University, United States

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
19 April 2023
Verified
Conference Paper
Open access
ÁCÁngel Alexander Cabrera
Ángel Alexander Cabrera
EFErica Fu
Erica Fu
DBDonald Bertucci
Donald Bertucci
KHKenneth Holstein
Kenneth Holstein
ATAmeet Talwalkar
Ameet Talwalkar
APAdam Perer
Adam Perer
Machine learning models with high accuracy on test data can still produce systematic failures, such as harmful biases and safety issues, when deployed in the real world. To detect and mitigate such failures, practitioners run behavioral evaluation of their models, checking model outputs for specific types of inputs. Behavioral evaluation is important but challenging, requiring that practitioners discover real-world patterns and validate systematic failures. We conducted 18 semi-structured interviews with ML practitioners to better understand the challenges of behavioral evaluation and found that it is a collaborative, use-case-first process that is not adequately supported by existing task- and domain-specific tools. Using these findings, we designed zeno, a general-purpose framework for visualizing and testing AI systems across diverse use cases. In four case studies with participants using zeno on real-world models, we found that practitioners were able to reproduce previous manual analyses and discover new systematic failures.