Hyo Jin Do

Hyo Jin Do

Auto-generated profile, based on publicly available information

Hyo Jin Do

IBM Research, Cambridge, MA, USA

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
14 April 2023 in Proceedings of the ACM on Human-Computer Interaction
Verified
Journal Article
Open access
HDHyo Jin Do
Hyo Jin Do
HKHa-Kyung Kong
Ha-Kyung Kong
PTPooja Tetali
Pooja Tetali
JLJaewook Lee
Jaewook Lee
BBBrian P. Bailey
Brian P. Bailey
Conversational agents (CAs) can analyze online conversations using natural language techniques and effectively facilitate group discussions by sending supervisory messages. However, if a CA makes imperfect interventions, users may stop trusting the CA and discontinue using it. In this study, we demonstrate how inaccurate interventions of a CA and a conversational repair strategy can influence user acceptance of the CA, members' participation in the discussion, perceived discussion experience between the members, and group performance. We built a CA that encourages the participation of members with low contributions in an online chat discussion in which a small group (3-6 members) performs a decision-making task. Two types of errors can occur when detecting under-contributing members: 1) false-positive (FP) errors happen when the CA falsely identifies a member as under-contributing and 2) false-negative (FN) errors occur when the CA misses detecting an under-contributing member. We designed a conversational repair strategy that gives users a chance to contest the detection results and the agent sends a correctional message if an error is detected. Through an online study with 175 participants, we found that participants who received FN error messages reported higher acceptance of the CA and better discussion experience, but participated less compared to those who received FP error messages. The conversational repair strategy moderated the effect of errors such as improving the perceived discussion experience of participants who received FP error messages. Based on our findings, we offer design implications for which model should be selected by practitioners between high precision (i.e., fewer FP errors) and high recall (i.e., fewer FN errors) models depending on the desired effects. When frequent FP errors are expected, we suggest using the conversational repair strategy to improve the perceived discussion experience.

27 March 2023
Verified
Conference Paper
Open access
MBMichelle Brachman
Michelle Brachman
QPQian Pan
Qian Pan
HDHyo Jin Do
Hyo Jin Do
CDCasey Dugan
Casey Dugan
ACArunima Chaudhary
Arunima Chaudhary
GTGegi Thomas
Gegi Thomas
While natural language systems continue improving, they are still imperfect. If a user has a better understanding of how a system works, they may be able to better accomplish their goals even in imperfect systems. We explored whether explanations can support effective authoring of natural language utterances and how those explanations impact users’ mental models in the context of a natural language system that generates small programs. Through an online study (n=252), we compared two main types of explanations: 1) system-focused, which provide information about how the system processes utterances and matches terms to a knowledge base, and 2) social, which provide information about how other users have successfully interacted with the system. Our results indicate that providing social suggestions of terms to add to an utterance helped users to repair and generate correct flows more than system-focused explanations or social recommendations of words to modify. We also found that participants commonly understood some mechanisms of the natural language system, such as the matching of terms to a knowledge base, but they often lacked other critical knowledge, such as how the system handled structuring and ordering. Based on these findings, we make design recommendations for supporting interactions with and understanding of natural language systems.

28 September 2023 in Proceedings of the ACM on Human-Computer Interaction
Verified
Journal Article
Open access
HDHyo Jin Do
Hyo Jin Do
HKHa-Kyung Kong
Ha-Kyung Kong
PTPooja Tetali
Pooja Tetali
KKKarrie Karahalios
Karrie Karahalios
BBBrian P. Bailey
Brian P. Bailey
A conversational agent (CA) effectively facilitates online group discussions at scale. However, users may have expectations about how well the CA would perform that do not match with the actual performance, compromising technology acceptance. We built a facilitator CA that detects a member who has low contribution during a synchronous group chat discussion and asks the person to participate more. We designed three techniques to set end-user expectations about how accurately the CA identifies an under-contributing member: 1)information: explicitly communicating the accuracy of the detection algorithm, 2)explanation: providing an overview of the algorithm and the data used for the detection, and 3)adjustment: enabling users to gain a feeling of control over the algorithm. We conducted an online experiment with 163 crowdworkers in which each group completed a collaborative decision-making task and experienced one of the techniques. Through surveys and interviews, we found that the explanation technique was the most effective strategy overall as it reduced user embarrassment, increased the perceived intelligence of the CA, and helped users better understand the detection algorithm. In contrast, the information technique reduced members' contributions and the adjustment technique led to a more negative perceived discussion experience. We also discovered that the interactions with other team members diluted the effects of the techniques on users' performance expectations and acceptance of the CA. We discuss implications for better designing expectation-setting techniques for AI-team collaboration such as ways to improve collaborative decision outcomes and quality of contributions.

22 June 2025
Verified
Conference Paper
Open access
HDHyo Jin Do
Hyo Jin Do
MFMolly Q Feldman
Molly Q Feldman
JHJessica He
Jessica He
AHAngel Hsing-Chi Hwang
Angel Hsing-Chi Hwang
SKSeyun Kim
Seyun Kim

7 November 2024 in Proceedings of the ACM on Human-Computer Interaction
Verified
Journal Article
Open access
HDHyo Jin Do
Hyo Jin Do
MBMichelle Brachman
Michelle Brachman
CDCasey Dugan
Casey Dugan
QPQian Pan
Qian Pan
PRPriyanshu Rai
Priyanshu Rai
RTRoshni Thawani
Roshni Thawani
Forming accurate mental models that align with the actual behavior of an AI system is critical for successful user experience and interactions. One way to develop mental models is through information shared by other users. However, this social information can be inaccurate and there is a lack of research examining whether inaccurate social information influences the development of accurate mental models. To address this gap, our study investigates the impact of social information accuracy on mental models, as well as whether prompting users to validate the social information can mitigate the impact. We conducted a between-subject experiment with 39 crowdworkers where each participant interacted with our AI system that automates a workflow given a natural language sentence. We compared participants' mental models between those exposed to social information of how the AI system worked, both correct and incorrect, versus those who formed mental models through their own usage of the system. Specifically, we designed three experimental conditions: 1) validation condition that presented the social information followed by an opportunity to validate its accuracy through testing example utterances, 2) social information condition that presented the social information only, without the validation opportunity, and 3) control condition that allowed users to interact with the system without any social information. Our results revealed that the inclusion of the validation process had a positive impact on the development of accurate mental models, especially around the knowledge distribution aspect of mental models. Furthermore, participants were more willing to share comments with others when they had the chance to validate the social information. The impact of inaccurate social information on altering user mental models was found to be non-significant, while 69.23% of participants incorrectly judged the social information accuracy at least once. We discuss the implications of these findings for designing tools that support the validation of social information and thereby improve human-AI interactions.

27 September 2025
Verified
Conference Paper
Open access
ZAZahra Ashktorab
Zahra Ashktorab
MDMichael Desmond
Michael Desmond
QPQian Pan
Qian Pan
JJJames M. Johnson
James M. Johnson
MCMartin Santillán Cooper
Martin Santillán Cooper
WGWerner Geyer
Werner Geyer