Yu Huang

Yu Huang

Auto-generated profile, based on publicly available information

Yu Huang

Vanderbilt University, Nashville, TN, USA

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
14 April 2024
Verified
Conference Paper
Open access
YXYuankai Xue
Yuankai Xue
HCHanlin Chen
Hanlin Chen
GBGina R. Bai
Gina R. Bai
RTRobert Tairas
Robert Tairas
YHYu Huang
Yu Huang
Generative AI, notably ChatGPT, has garnered attention in computer science education. This paper presents a controlled experiment that explores ChatGPT's role in CS1 in a classroom setting. Specifically, we aim to investigate the impact of ChatGPT on student learning outcomes and their behaviors when working on programming assignments. Participants were tasked with creating a UML diagram and subsequently implementing its design through programming, followed by a closed-book post-evaluation and a post-survey. All the participants were required to screen-record the whole process. In total, 56 participants were recruited, with 48 successful screen recordings. Participants in the Experimental Group can access ChatGPT 3.5 and other online resources, such as Google and Stack Overflow when creating the UML diagram and programming; however, participants in the Control Group can access all online resources except for ChatGPT (i.e., the only design variable is the access to ChatGPT). Finally, we measured and analyzed participants' learning outcomes through their UML diagram, programming, and post-evaluation scores. We also analyzed the time participants took to complete the tasks and their interactions with ChatGPT and other resources from the screen recordings. After finishing the tasks, student participants also provided their perceptions of using ChatGPT in CS1 through a post-survey. With rigorous quantitative and qualitative analysis, we found that (1) using ChatGPT does not present a significant impact on students' learning performance in the CS1 assignment-style tasks; (2) once using ChatGPT, students' tendency to explore other traditional educational resources is largely reduced (though available) and they tend to rely solely on ChatGPT, and this reliance on ChatGPT did not guarantee enhanced learning performance; (3) the majority of students hold neutral views on ChatGPT's role in CS1 programming but most of them raised concerns about its potential ethical issues and inconsistent performance across different tasks. We hope this study can help educators and students better understand the impact of ChatGPT in CS1 and inspire future work to provide proper guidelines for using ChatGPT in introductory programming classes.

24 May 2023 in IEEE Transactions on Software Engineering
Verified
Journal Article
Closed access
ABAakash Bansal
Aakash Bansal
ZEZachary Eberhart
Zachary Eberhart
ZKZachary Karas
Zachary Karas
YHYu Huang
Yu Huang
CMCollin McMillan
Collin McMillan
Source code summarization is the task of writing natural language descriptions of source code. The primary use of these descriptions is in documentation for programmers. Automatic generation of these descriptions is a high value research target due to the time cost to programmers of writing these descriptions themselves. In recent years, a confluence of software engineering and artificial intelligence research has made inroads into automatic source code summarization through applications of neural models of that source code. However, an Achilles' heel to a vast majority of approaches is that they tend to rely solely on the context provided by the source code being summarized. But empirical studies in program comprehension are quite clear that the information needed to describe code much more often resides in the context in the form of Function Call Graph surrounding that code. In this paper, we present a technique for encoding this call graph context for neural models of code summarization. We implement our approach as a supplement to existing approaches, and show statistically significant improvement over existing approaches. In a human study with 20 programmers, we show that programmers perceive generated summaries to generally be as accurate, readable, and concise as human-written summaries.

12 July 2024 in Proceedings of the ACM on Software Engineering
Verified
Journal Article
Open access
YZYifan Zhang
Yifan Zhang
JLJiliang Li
Jiliang Li
ZKZachary Karas
Zachary Karas
ABAakash Bansal
Aakash Bansal
TLToby Jia-Jun Li
Toby Jia-Jun Li
YHYu Huang
Yu Huang
Neural code summarization leverages deep learning models to automatically generate brief natural language summaries of code snippets. The development of Transformer models has led to extensive use of attention during model design. While existing work has primarily and almost exclusively focused on static properties of source code and related structural representations like the Abstract Syntax Tree (AST), few studies have considered human attention — that is, where programmers focus while examining and comprehending code. In this paper, we develop a method for incorporating human attention into machine attention to enhance neural code summarization. To facilitate this incorporation and vindicate this hypothesis, we introduce E ye T rans , which consists of three steps: (1) we conduct an extensive eye-tracking human study to collect and pre-analyze data for model training, (2) we devise a data-centric approach to integrate human attention with machine attention in the Transformer architecture, and (3) we conduct comprehensive experiments on two code summarization tasks to demonstrate the effectiveness of incorporating human attention into Transformers. Integrating human attention leads to an improvement of up to 29.91% in Functional Summarization and up to 6.39% in General Code Summarization performance, demonstrating the substantial benefits of this combination. We further explore performance in terms of robustness and efficiency by creating challenging summarization scenarios in which E ye Trans exhibits interesting properties. We also visualize the attention map to depict the simplifying effect of machine attention in the Transformer by incorporating human attention. This work has the potential to propel AI research in software engineering by introducing more human-centered approaches and data.

26 August 2024 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
ZKZachary Karas
Zachary Karas
ABAakash Bansal
Aakash Bansal
YZYifan Zhang
Yifan Zhang
TLToby Li
Toby Li
CMCollin McMillan
Collin McMillan
YHYu Huang
Yu Huang
Code summarization is the task of creating short, natural language descriptions of source code. It is an important part of code comprehension and a powerful method of documentation. Previous work has made progress in identifying where programmers focus in code as they write their own summaries (i.e., Writing). However, there is currently a gap in studying programmers’ attention as they read code with pre-written summaries (i.e., Reading). As a result, it is currently unknown how these two forms of code comprehension compare: Reading and Writing. Also, there is a limited understanding of programmer attention with respect to program semantics. We address these shortcomings with a human eye-tracking study ( n = 27) comparing Reading and Writing. We examined programmers’ attention with respect to fine-grained program semantics, including their attention sequences (i.e., scan paths). We find distinctions in programmer attention across the comprehension tasks, similarities in reading patterns between them, and differences mediated by demographic factors. This can help guide code comprehension in both computer science education and automated code summarization. Furthermore, we mapped programmers’ gaze data onto the Abstract Syntax Tree to explore another representation of human attention. We find that visual behavior on this structure is not always consistent with that on source code.

15 April 2024
Verified
Conference Paper
Open access
JLJiliang Li
Jiliang Li
YZYifan Zhang
Yifan Zhang
ZKZachary Karas
Zachary Karas
CMCollin McMillan
Collin McMillan
KLKevin Leach
Kevin Leach
YHYu Huang
Yu Huang
Recent language models have demonstrated proficiency in summarizing source code. However, as in many other domains of machine learning, language models of code lack sufficient explainability --- informally, we lack a formulaic or intuitive understanding of what and how models learn from code. Explainability of language models can be partially provided if, as the models learn to produce higher-quality code summaries, they also align in deeming the same code parts important as those identified by human programmers. In this paper, we report negative results from our investigation of explainability of language models in code summarization through the lens of human comprehension. We measure human focus on code using eye-tracking metrics such as fixation counts and duration in code summarization tasks. To approximate language model focus, we employ a state-of-the-art model-agnostic, black-box, perturbation-based approach, SHAP (SHapley Additive exPlanations), to identify which code tokens influence that generation of summaries. Using these settings, we find no statistically significant relationship between language models' focus and human programmers' attention. Furthermore, alignment between model and human foci in this setting does not seem to dictate the quality of the LLM-generated summaries. Our study highlights an inability to align human focus with SHAP-based model focus measures. This result calls for future investigation of multiple open questions for explainable language models for code summarization and software engineering tasks in general, including the training mechanisms of language models for code, whether there is an alignment between human and model attention on code, whether human attention can improve the development of language models, and what other model focus measures are appropriate for improving explainability.

26 March 2025 in IEEE Transactions on Software Engineering
Verified
Journal Article
Closed access
RWRobert Wallace
Robert Wallace
ABAakash Bansal
Aakash Bansal
ZKZachary Karas
Zachary Karas
NTNingzhi Tang
Ningzhi Tang
YHYu Huang
Yu Huang
CMCollin McMillan
Collin McMillan
Programmer attention represents the visual focus of programmers on parts of the source code in pursuit of programming tasks. The focus of current research in modeling this programmer attention has been on using mouse cursors, keystrokes, or eye tracking equipment to map areas in a snippet of code. These approaches have traditionally only mapped attention for a single method. However, there is a knowledge gap in the literature because programming tasks such as source code summarization require programmers to use contextual knowledge that can only be found in other parts of the project, not only in a single method. To address this knowledge gap, we conducted an in-depth human study with 10 Java programmers, where each programmer generated summaries for 40 methods from five large Java projects over five one-hour sessions. We used eye tracking equipment to map the visual attention of programmers while they wrote the summaries. We also rate the quality of each summary. We found eye-gaze patterns and metrics that define common behaviors between programmer attention during context-aware code summarization. Specifically, we found that programmers need to read up to 35% fewer words (p<0.01) over the whole session, and revisit 13% fewer words (p<0.03) as they summarize each method during a session, while maintaining the quality of summaries. We also found that the amount of source code a participant looks at correlates with a higher quality summary, but this trend follows a bell-shaped curve, such that after a threshold reading more source code leads to a significant decrease (p<0.01) in the quality of summaries. We also gathered insight into the type of methods in the project that provide the most contextual information for code summarization based on programmer attention. Specifically, we observed that programmers spent a majority of their time looking at methods inside the same class as the target method to be summarized. Surprisingly, we found that programmers spent significantly less time looking at methods in the call graph of the target method. We discuss how our empirical observations may aid future studies towards modeling programmer attention and improving context-aware automatic source code summarization.