Crystal Qian

Crystal Qian

Auto-generated profile, based on publicly available information

Crystal Qian

Harvard University MIT & NBER MIT

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications

11 May 2024
Verified
Conference Paper
Open access
EREmily Reif
Emily Reif
CQCrystal Qian
Crystal Qian
JWJames Wexler
James Wexler
MKMinsuk Kahng
Minsuk Kahng
Making sense of unstructured text datasets is perennially difficult, yet increasingly relevant with Large Language Models. Data practitioners often rely on dataset summaries, especially distributions of various derived features. Some features, like toxicity or topics, are relevant to many datasets, but many interesting features are domain specific: instruments and genres for a music dataset, or diseases and symptoms for a medical dataset. Accordingly, data practitioners often run custom analyses for each dataset, which is cumbersome and difficult, or use unsupervised methods. We present AutoHistograms, a visualization tool leveraging LLMs. AutoHistograms automatically identifies relevant entity-based features, visualizes them, and allows the user to interactively query the dataset for new categories of entities. In a user study with (n=10) data practitioners, we observe that participants were able to quickly onboard to AutoHistograms, use the tool to identify actionable insights, and conceptualize a broad range of applicable use cases. Together, this tool and user study contribute to the growing field of LLM-assisted sensemaking tools.

2 May 2024
Verified
Conference Paper
Open access
CQCrystal Qian
Crystal Qian
EREmily Reif
Emily Reif
MKMinsuk Kahng
Minsuk Kahng
As large language models (LLMs) become more advanced and impactful, it is increasingly important to scrutinize the data that they rely upon and produce. What is it to be a dataset practitioner doing this work? We approach this in two parts: first, we define the role of “dataset practitioners” by performing a retrospective analysis on the responsibilities of teams contributing to LLM development at a technology company, Google. Then, we conduct semi-structured interviews with a cross-section of these practitioners (N=10). We find that although data quality is a top priority, there is little consensus around what data quality is and how to evaluate it. Consequently, practitioners either rely on their own intuition or write custom code to evaluate their data. We discuss potential reasons for this phenomenon and opportunities for alignment.

10 July 2023
Verified
Conference Paper
Open access
NANarges Ashtari
Narges Ashtari
RMRyan Mullins
Ryan Mullins
CQCrystal Qian
Crystal Qian
JWJames Wexler
James Wexler
ITIan Tenney
Ian Tenney
MPMahima Pushkarna
Mahima Pushkarna
Models are interpretable when machine learning (ML) practitioners can readily understand the reasoning behind their predictions. Ironically, little is known about the ML practitioners’ experience of discovering and adopting novel interpretability techniques in production settings. In a qualitative study with 18 practitioners at a large technology company working with text data, we found that despite varied tasks, practitioners experienced nearly identical challenges related to interpretability methods in model analysis workflows. These stem from problem formulation, the social nature of interpretability investigations, and non-standard practices in cross-functional organizational contexts. A follow-up examination of early-stage design probes with seven practitioners suggests that self-reported experts are “perpetual intermediates”, who can benefit from regular, responsive, and in-situ education about interpretability methods across workflows, regardless of prior experience with models, analysis tools, or interpretability techniques. From these findings, we emphasize the need for multi-stage support for learning of interpretability methods for real-world NLP applications.

25 April 2025
Verified
Conference Paper
Open access
CQCrystal Qian
Crystal Qian
MLMichael Xieyang Liu
Michael Xieyang Liu
EREmily Reif
Emily Reif
GSGrady Simon
Grady Simon
NHNada Hussein
Nada Hussein
MKMinsuk Kahng
Minsuk Kahng

9 October 2024 in arXiv
Verified
Preprint
Open access
PDF file
BGBasile Garcia
Basile Garcia
CQCrystal Qian
Crystal Qian
SPStefano Palminteri
Stefano Palminteri
As large language models (LLMs) become increasingly integrated into society, their alignment with human morals is crucial. To better understand this alignment, we created a large corpus of human- and LLM-generated responses to various moral scenarios. We found a misalignment between human and LLM moral assessments; although both LLMs and humans tended to reject morally complex utilitarian dilemmas, LLMs were more sensitive to personal framing. We then conducted a quantitative user study involving 230 participants (N=230), who evaluated these responses by determining whether they were AI-generated and assessed their agreement with the responses. Human evaluators preferred LLMs' assessments in moral scenarios, though a systematic anti-AI bias was observed: participants were less likely to agree with judgments they believed to be machine-generated. Statistical and NLP-based analyses revealed subtle linguistic differences in responses, influencing detection and agreement. Overall, our findings highlight the complexities of human-AI perception in morally charged decision-making.