James M. Johnson

James M. Johnson

Auto-generated profile, based on publicly available information

James M. Johnson

IBM Research, Cambridge, Massachusetts, USA

Research Focus & Metrics

Academic Career Length

33Years (1994-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
14 April 2023 in Proceedings of the ACM on Human-Computer Interaction
Verified
Journal Article
Open access
IMImani Munyaka
Imani Munyaka
ZAZahra Ashktorab
Zahra Ashktorab
CDCasey Dugan
Casey Dugan
JJJ. Johnson
J. Johnson
QPQian Pan
Qian Pan
Human-AI teams are increasingly prevalent in various domains. We investigate how the decision-making of a team member in a human-AI team impacts the outcome of the collaboration and perceived team-efficacy. In a large scale study on Mechanical Turk (n=125), we find significant differences across different decision making styles and disclosed AI identity disclosure in an AI-driven collaborative game. We find that autocratic decision-making negatively impacts team-efficacy in Human-AI teams, similar to its effects on human-only teams. We find that decision making style and AI-identity disclosure impacts how individuals make decisions in a collaborative context. We discuss our findings of the differences of collaborative behavior in human-human-AI teams and human-AI-AI teams.

18 March 2024
Verified
Conference Paper
Open access
MDMichael Desmond
Michael Desmond
ZAZahra Ashktorab
Zahra Ashktorab
QPQian Pan
Qian Pan
CDCasey Dugan
Casey Dugan
JJJames M. Johnson
James M. Johnson
With the rapid improvement in large language model (LLM) capabilities, its becoming more difficult to measure the quality of outputs generated by natural language generation (NLG) systems. Conventional metrics such as BLEU and ROUGE are bound to reference data, and are generally unsuitable for tasks that require creative or diverse outputs. Human evaluation is an option, but manually evaluating generated text is difficult to do well, and expensive to scale and repeat as requirements and quality criteria change. Recent work has focused on the use of LLMs as customize-able NLG evaluators, and initial results are promising. In this demonstration we present EvaluLLM, an application designed to help practitioners setup, run and review evaluation over sets of NLG outputs, using an LLM as a custom evaluator. Evaluation is formulated as a series of choices between pairs of generated outputs conditioned on a user provided evaluation criteria. This approach simplifies the evaluation task and obviates the need for complex scoring algorithms. The system can be applied to general evaluation, human assisted evaluation, and model selection problems.

27 March 2023
Verified
Conference Paper
Open access
MBMichelle Brachman
Michelle Brachman
QPQian Pan
Qian Pan
HDHyo Jin Do
Hyo Jin Do
CDCasey Dugan
Casey Dugan
ACArunima Chaudhary
Arunima Chaudhary
GTGegi Thomas
Gegi Thomas
While natural language systems continue improving, they are still imperfect. If a user has a better understanding of how a system works, they may be able to better accomplish their goals even in imperfect systems. We explored whether explanations can support effective authoring of natural language utterances and how those explanations impact users’ mental models in the context of a natural language system that generates small programs. Through an online study (n=252), we compared two main types of explanations: 1) system-focused, which provide information about how the system processes utterances and matches terms to a knowledge base, and 2) social, which provide information about how other users have successfully interacted with the system. Our results indicate that providing social suggestions of terms to add to an utterance helped users to repair and generate correct flows more than system-focused explanations or social recommendations of words to modify. We also found that participants commonly understood some mechanisms of the natural language system, such as the matching of terms to a knowledge base, but they often lacked other critical knowledge, such as how the system handled structuring and ordering. Based on these findings, we make design recommendations for supporting interactions with and understanding of natural language systems.


1 March 2016
Verified
Conference Paper
Closed access
JHJairus Hihn
Jairus Hihn
LJLeora Juster
Leora Juster
JJJames Johnson
James Johnson
TMTim Menzies
Tim Menzies
GMGeorge Michael
George Michael
Estimators and analysts are increasingly being tasked to develop better models and reliable cost estimates in support of program planning and execution. While there has been extensive work on improving parametric methods for cost estimation, there is very little focus on the use of cost models based on analogy and clustering algorithms. In this paper we summarize the results of our research in developing an analogy method for estimating NASA spacecraft flight software using spectral clustering on system characteristics (symbolic non-numerical data) and evaluate its performance by comparing it to a number of the most commonly used estimation methods. The strengths and weaknesses of each method based on their performance are also discussed. The paper concludes with an overview of the analogy estimation tool (ASCoT) developed for use within NASA that implements the recommended analogy algorithm.

7 November 2024 in Proceedings of the ACM on Human-Computer Interaction
Verified
Journal Article
Open access
HDHyo Jin Do
Hyo Jin Do
MBMichelle Brachman
Michelle Brachman
CDCasey Dugan
Casey Dugan
QPQian Pan
Qian Pan
PRPriyanshu Rai
Priyanshu Rai
RTRoshni Thawani
Roshni Thawani
Forming accurate mental models that align with the actual behavior of an AI system is critical for successful user experience and interactions. One way to develop mental models is through information shared by other users. However, this social information can be inaccurate and there is a lack of research examining whether inaccurate social information influences the development of accurate mental models. To address this gap, our study investigates the impact of social information accuracy on mental models, as well as whether prompting users to validate the social information can mitigate the impact. We conducted a between-subject experiment with 39 crowdworkers where each participant interacted with our AI system that automates a workflow given a natural language sentence. We compared participants' mental models between those exposed to social information of how the AI system worked, both correct and incorrect, versus those who formed mental models through their own usage of the system. Specifically, we designed three experimental conditions: 1) validation condition that presented the social information followed by an opportunity to validate its accuracy through testing example utterances, 2) social information condition that presented the social information only, without the validation opportunity, and 3) control condition that allowed users to interact with the system without any social information. Our results revealed that the inclusion of the validation process had a positive impact on the development of accurate mental models, especially around the knowledge distribution aspect of mental models. Furthermore, participants were more willing to share comments with others when they had the chance to validate the social information. The impact of inaccurate social information on altering user mental models was found to be non-significant, while 69.23% of participants incorrectly judged the social information accuracy at least once. We discuss the implications of these findings for designing tools that support the validation of social information and thereby improve human-AI interactions.