Casey Dugan

Casey Dugan

Auto-generated profile, based on publicly available information

Casey Dugan

IBM Research, Cambridge, MA, USA

Research Focus & Metrics

Academic Career Length

19Years (2008-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
2 May 2025 in Proceedings of the ACM on Human-Computer Interaction
Verified
Journal Article
Open access
SHShang-Ling Hsu
Shang-Ling Hsu
RSRaj Sanjay Shah
Raj Sanjay Shah
PSPrathik Senthil
Prathik Senthil
ZAZahra Ashktorab
Zahra Ashktorab
CDCasey Dugan
Casey Dugan
DYDiyi Yang
Diyi Yang
Millions of users come to online peer counseling platforms to seek support. However, studies show that online peer support groups are not always as effective as expected, largely due to users' negative experiences with unhelpful counselors. Peer counselors are key to the success of online peer counseling platforms, but most often do not receive appropriate training. Hence, we introduce CARE: an AI-based tool to empower and train peer counselors through practice and feedback. Concretely, CARE helps diagnose which counseling strategies are needed in a given situation and suggests example responses to counselors during their practice sessions. Building upon the Motivational Interviewing framework, CARE utilizes large-scale counseling conversation data with text generation techniques to enable these functionalities. We demonstrate the efficacy of CARE by performing quantitative evaluations and qualitative user studies through simulated chats and semi-structured interviews, finding that CARE especially helps novice counselors in challenging situations. The code is available at https://github.com/SALT-NLP/CARE.

14 April 2023 in Proceedings of the ACM on Human-Computer Interaction
Verified
Journal Article
Open access
IMImani Munyaka
Imani Munyaka
ZAZahra Ashktorab
Zahra Ashktorab
CDCasey Dugan
Casey Dugan
JJJ. Johnson
J. Johnson
QPQian Pan
Qian Pan
Human-AI teams are increasingly prevalent in various domains. We investigate how the decision-making of a team member in a human-AI team impacts the outcome of the collaboration and perceived team-efficacy. In a large scale study on Mechanical Turk (n=125), we find significant differences across different decision making styles and disclosed AI identity disclosure in an AI-driven collaborative game. We find that autocratic decision-making negatively impacts team-efficacy in Human-AI teams, similar to its effects on human-only teams. We find that decision making style and AI-identity disclosure impacts how individuals make decisions in a collaborative context. We discuss our findings of the differences of collaborative behavior in human-human-AI teams and human-AI-AI teams.

11 May 2024
Verified
Conference Paper
Open access
MBMichelle Brachman
Michelle Brachman
AEAmina El-Ashry
Amina El-Ashry
CDCasey Dugan
Casey Dugan
WGWerner Geyer
Werner Geyer
Large Language Models (LLMs) have introduced a paradigm shift in interaction with AI technology, enabling knowledge workers to complete tasks by specifying their desired outcome in natural language. LLMs have the potential to increase productivity and reduce tedious tasks in an unprecedented way. A systematic study of LLM adoption for work can provide insight into how LLMs can best support these workers. To explore knowledge workers’ current and desired usage of LLMs, we ran a survey (n=216). Workers described tasks they already used LLMs for, like generating code or improving text, but imagined a future with LLMs integrated into their workflows and data. We discuss implications for adoption and design of generative AI technologies for knowledge work.

18 March 2024
Verified
Conference Paper
Open access
MDMichael Desmond
Michael Desmond
ZAZahra Ashktorab
Zahra Ashktorab
QPQian Pan
Qian Pan
CDCasey Dugan
Casey Dugan
JJJames M. Johnson
James M. Johnson
With the rapid improvement in large language model (LLM) capabilities, its becoming more difficult to measure the quality of outputs generated by natural language generation (NLG) systems. Conventional metrics such as BLEU and ROUGE are bound to reference data, and are generally unsuitable for tasks that require creative or diverse outputs. Human evaluation is an option, but manually evaluating generated text is difficult to do well, and expensive to scale and repeat as requirements and quality criteria change. Recent work has focused on the use of LLMs as customize-able NLG evaluators, and initial results are promising. In this demonstration we present EvaluLLM, an application designed to help practitioners setup, run and review evaluation over sets of NLG outputs, using an LLM as a custom evaluator. Evaluation is formulated as a series of choices between pairs of generated outputs conditioned on a user provided evaluation criteria. This approach simplifies the evaluation task and obviates the need for complex scoring algorithms. The system can be applied to general evaluation, human assisted evaluation, and model selection problems.

16 October 2025 in Proceedings of the ACM on Human-Computer Interaction
Verified
Journal Article
Open access
MBMichelle Brachman
Michelle Brachman
AEAmina El-Ashry
Amina El-Ashry
CDCasey Dugan
Casey Dugan
WGWerner Geyer
Werner Geyer
Large Language Models (LLMs) have introduced a paradigm shift in interaction with AI technology, enabling knowledge workers to complete tasks by specifying their desired outcome in natural language. LLMs have the potential to increase productivity and reduce tedious tasks in an unprecedented way. A systematic study of LLM adoption for work can provide insight into how LLMs can best support these workers. To explore knowledge workers' current and desired usage of LLMs, we ran a survey (n=216). Workers described tasks they already used LLMs for, like generating code or improving text, but imagined a future with LLMs integrated into their workflows and data. We ran a second survey (n=107) a year later that validated our initial findings and provides insight into up-to-date LLM use by knowledge workers. We discuss implications for adoption and design of generative AI technologies for knowledge work.

19 April 2023
Verified
Conference Paper
Open access
ZAZahra Ashktorab
Zahra Ashktorab
BHBenjamin Hoover
Benjamin Hoover
MAMayank Agarwal
Mayank Agarwal
CDCasey Dugan
Casey Dugan
WGWerner Geyer
Werner Geyer
MYMikhail Yurochkin
Mikhail Yurochkin
Mitigating algorithmic bias is a critical task in the development and deployment of machine learning models. While several toolkits exist to aid machine learning practitioners in addressing fairness issues, little is known about the strategies practitioners employ to evaluate model fairness and what factors influence their assessment, particularly in the context of text classification. Two common approaches of evaluating the fairness of a model are group fairness and individual fairness. We run a study with Machine Learning practitioners (n=24) to understand the strategies used to evaluate models. Metrics presented to practitioners (group vs. individual fairness) impact which models they consider fair. Participants focused on risks associated with underpredicting / overpredicting and model sensitivity relative to identity token manipulations. We discover fairness assessment strategies involving personal experiences or how users form groups of identity tokens to test model fairness. We provide recommendations for interactive tools for evaluating fairness in text classification.