Zengyang Li

Zengyang Li

Auto-generated profile, based on publicly available information

Zengyang Li

School of Computer Science, Central China Normal University, WuHan, HuBei, China

Research Focus & Metrics

Academic Career Length

9Years (2018-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
6 October 2025 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
YFYujia Fu
Yujia Fu
PLPeng Liang
Peng Liang
ATAmjed Tahir
Amjed Tahir
ZLZengyang Li
Zengyang Li
MSMojtaba Shahin
Mojtaba Shahin
JCJinfu Chen
Jinfu Chen
Modern code generation tools utilizing AI models like Large Language Models have gained increased popularity due to their ability to produce functional code. However, their usage presents security challenges, often resulting in insecure code merging into the code base. Thus, evaluating the quality of generated code, especially its security, is crucial. While prior research explored various aspects of code generation, the focus on security has been limited, mostly examining code produced in controlled environments rather than open source development scenarios. To address this gap, we conducted an empirical study, analyzing code snippets generated by GitHub Copilot and two other AI code generation tools (i.e., CodeWhisperer and Codeium) from GitHub projects. Our analysis identified 733 snippets, revealing a high likelihood of security weaknesses, with 29.5% of Python and 24.2% of JavaScript snippets affected. These issues span 43 Common Weakness Enumeration (CWE) categories, including significant ones like CWE-330: Use of Insufficiently Random Values , CWE-94: Improper Control of Generation of Code , and CWE-79: Cross-site Scripting . Notably, eight of those CWEs are among the 2023 CWE Top-25, highlighting their severity. We further examined using Copilot Chat to fix security issues in Copilot-generated code by providing Copilot Chat with warning messages from the static analysis tools, and up to 55.5% of the security issues can be fixed. We finally provide the suggestions for mitigating security issues in generated code.

2 September 2024 in Journal of Systems and Software
Verified
Journal Article
Closed access
XZXiyu Zhou
Xiyu Zhou
PLPeng Liang
Peng Liang
BZBeiqi Zhang
Beiqi Zhang
ZLZengyang Li
Zengyang Li
AAAakash Ahmad
Aakash Ahmad
MWMuhammad Waseem
Muhammad Waseem

20 April 2025 in Journal of Systems and Software
Verified
Journal Article
Closed access
YCYangxiao Cai
Yangxiao Cai
PLPeng Liang
Peng Liang
YWYifei Wang
Yifei Wang
ZLZengyang Li
Zengyang Li
MSMojtaba Shahin
Mojtaba Shahin

27 January 2025 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
RMRan Mo
Ran Mo
DWDongyu Wang
Dongyu Wang
WZWenjing Zhan
Wenjing Zhan
YJYingjie Jiang
Yingjie Jiang
YWYepeng Wang
Yepeng Wang
YMYutao Ma
Yutao Ma
AI programming has become a popular topic in recent years. Code suggestion, with code suggestion being a key capability of AI programming. Copilot, an “AI programmer” that provides code suggestions from natural language descriptions, has been launched by GitHub and OpenAI. By far, Copilot has been widely used by millions of developers. However, little work has systematically evaluated the correctness of Copilot's suggestions. We conducted an empirical study on all 2,033 LeetCode problems to assess Copilot's code generation across four mainstream languages: C, Java, JavaScript, and Python. We have found that: 1) 70.0% of problems received at least one correct suggestion, with language-specific rates of 29.7% (C), 57.7% (Java), 54.1% (JavaScript), and 41.0% (Python); 2) Correctness decreases as problem difficulty increases, with acceptance rates of 89.3% (Easy), 72.1% (Medium), and 43.4% (Hard); 3) Acceptance rates vary across problem domains from 49.5% to 90.1%, while Graph problems challenge C and Python most, and Prefix Sum and Heap challenge Java and JavaScript most; 4) For the incorrect suggestions, we further summarize 17 types of error reasons accounting for their incorrectness and analyzed possible causes for why these errors occur. We believe our study can provide valuable insights into Copilot's capabilities and limitations.

15 December 2025 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
XZXiyu Zhou
Xiyu Zhou
RLRuiyin Li
Ruiyin Li
PLPeng Liang
Peng Liang
BZBeiqi Zhang
Beiqi Zhang
MSMojtaba Shahin
Mojtaba Shahin
CYChen Yang
Chen Yang
Design Rationale (DR) for software architecture decisions refers to the reasoning underlying architectural choices, which provides valuable insights into the different phases of the architecting process throughout software development. However, in practice, DR is often inadequately documented due to a lack of motivation and effort from developers. With the recent advancements in Large Language Models (LLMs), their capabilities in text comprehension, reasoning, and generation may enable the generation and recovery of DR for architecture decisions. In this study, we evaluated the performance of LLMs in generating DR for architecture decisions. First, we collected 50 Stack Overflow (SO) posts, 25 GitHub issues, and 25 GitHub discussions related to architecture decisions to construct a dataset of 100 architecture-related problems. Then, we selected five LLMs to generate DR for the architecture decisions with three prompting strategies, including zero-shot, chain of thought (CoT), and LLM-based agents. With the DR provided by human experts as ground truth, the Precision of LLM-generated DR with the three prompting strategies ranges from 0.267 to 0.278, Recall from 0.627 to 0.715, and F1-score from 0.351 to 0.389. Besides, 64.45% to 69.42% of the arguments of DR not mentioned by human experts are also helpful, 4.12% to 4.87% of the arguments have uncertain correctness, and 1.59% to 3.24% of the arguments are potentially misleading. To further understand the trustworthiness and applicability of LLM-generated DR in practice, we conducted semi-structured interviews with six practitioners. Based on the experimental and interview results, we discussed the pros and cons of the three prompting strategies, the strengths and limitations of LLM-generated DR, and the implications for the practical use of LLM-generated DR.

27 October 2024
Verified
Conference Paper
Open access
BZBeiqi Zhang
Beiqi Zhang
PLPeng Liang
Peng Liang
QFQiong Feng
Qiong Feng
YFYujia Fu
Yujia Fu
ZLZengyang Li
Zengyang Li
As one of the most popular dynamic languages, Python experiences a decrease in readability and maintainability when code smells are present. Recent advancements in Large Language Models have sparked growing interest in AI-enabled tools for both code generation and refactoring. GitHub Copilot is one such tool that has gained widespread usage. Copilot Chat, released in September 2023, functions as an interactive tool aimed at facilitating natural language-powered coding. However, limited attention has been given to understanding code smells in Copilot-generated Python code and Copilot Chat's ability to fix the code smells. To this end, we built a dataset comprising 102 code smells in Copilot-generated Python code. Our aim is to first explore the occurrence of code smells in Copilot-generated Python code and then evaluate the effectiveness of Copilot Chat in fixing these code smells employing different prompts. The results show that 8 out of 10 types of code smells can be detected in Copilot-generated Python code, among which Multiply-Nested Container is the most common one. For these code smells, Copilot Chat achieves a highest fixing rate of 87.1%, showing promise in fixing Python code smells generated by Copilot itself. In addition, the effectiveness of Copilot Chat in fixing these smells can be improved by providing more detailed prompts.