Kaiwen Ning

Kaiwen Ning

Auto-generated profile, based on publicly available information

Kaiwen Ning

Sun Yat-sen University and Pengcheng Laboratory, China

Research Focus & Metrics

Academic Career Length

3Years (2024-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
27 October 2024
Verified
Conference Paper
Open access
JCJiachi Chen
Jiachi Chen
QZQingyuan Zhong
Qingyuan Zhong
YWYanlin Wang
Yanlin Wang
KNKaiwen Ning
Kaiwen Ning
YLYongkun Liu
Yongkun Liu
ZZZibin Zheng
Zibin Zheng
Warning: Please note that this article contains potential harmful or offensive content. This content is only for the evaluating and analysis of LLMs and does not imply any intention to promote criminal activities. The emergence of Large Language Models (LLMs) has significantly influenced various aspects of software development activities. Despite their benefits, LLMs also pose notable risks, including the potential to generate harmful content and being abused by malicious developers to create malicious code. Several previous studies have focused on the ability of LLMs to resist the generation of harmful content that violates human ethical standards, such as biased or offensive content. However, there is no research evaluating the ability of LLMs to resist malicious code generation. To fill this gap, we propose RMCBench, the first benchmark comprising 473 prompts designed to assess the ability of LLMs to resist malicious code generation. This benchmark employs two scenarios: a text-to-code scenario, where LLMs are prompted with descriptions to generate code, and a code-to-code scenario, where LLMs translate or complete existing malicious code. Based on RMCBench, we conduct an empirical study on the 11 representative LLMs to assess their ability to resist malicious code generation. Our findings indicate that current LLMs have a limited ability to resist malicious code generation with an average refusal rate of 40.36% in text-to-code scenario and 11.52% in code-to-code scenario. The average refusal rate of all LLMs in RMCBench is only 28.71%; ChatGPT-4 has a refusal rate of only 35.73%. We also analyze the factors that affect LLM's ability to resist malicious code generation and provide implications for developers to enhance model robustness.

28 January 2026 in IEEE Transactions on Software Engineering
Verified
Journal Article
Closed access
KNKaiwen Ning
Kaiwen Ning
JCJiachi Chen
Jiachi Chen
JZJingwen Zhang
Jingwen Zhang
WLWei Li
Wei Li
ZWZexu Wang
Zexu Wang
ZZZibin Zheng
Zibin Zheng
Artificial intelligence (AI) agents are systems capable of perceiving their environment, autonomously planning and executing tasks. Recent advancements in Large Language Models (LLMs) have introduced a transformative paradigm for AI agents, enabling them to interact with external resources and tools through prompt techniques. This advancement has significantly extended the capabilities of LLMs, positioning LLM-based AI Agents as an important research area. In such agents, the workflow integrates developer-written code, which manages framework construction and logic control, with LLM-generated natural language that enhances dynamic decision-making and interaction. However, inconsistencies between LLM outputs and developer logic can lead to defects, such as tool invocation failures. These issues introduce specific risks, leading to various defects in LLM-based AI Agents, including service interruptions and incorrect output. Despite the importance of these issues, there is a lack of systematic work that focuses on analyzing LLM-based AI Agents to uncover defects in their code. To address this gap, we present the first study focused on identifying and detecting defects in LLM Agents. We collected and analyzed 14,754 relevant developer reports from StackOverflow and GitHub. We further filtered 2,604 valid posts to define and classify eight types of agent code defects. Then, we designed a static analysis tool, named Agentable, to detect these defects. Agentable leverages Code Property Graphs (CPGs) and LLMs to analyze Agent workflows by efficiently identifying specific code patterns and analyzing natural language descriptions. To evaluate Agentable, we constructed two datasets: AgentSet, which consists of 84 real world Agent projects, and AgentTest, which contains 78 Agent projects specifically designed to include various types of defects. Our evaluation shows that Agentable achieves a precision of 88.79% on the real-world agent dataset and a recall of 91.03% on the manually labeled defect dataset. Furthermore, our analysis identifies 889 defects in real-world agent projects, highlighting the prevalence of these issues in practice.

6 October 2026 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
KNKaiwen Ning
Kaiwen Ning
JCJiachi Chen
Jiachi Chen
QZQingyuan Zhong
Qingyuan Zhong
Tao ZhangTao Zhang
Tao Zhang
YWYanlin Wang
Yanlin Wang
ZZZibin Zheng
Zibin Zheng
With the advent of large language models (LLMs), numerous software service providers are developing LLMs tailored for code generation, such as CodeLlama. However, these models can be exploited by malicious developers to generate malicious code, posing severe threats to the software ecosystem. To address this issue, we first conducted an empirical study and built MCGTest , a dataset of \(406\) prompts designed to elicit malicious code from LLMs. Leveraging this dataset, we propose MCGMark , a watermarking method to trace and attribute LLM-generated malicious code. MCGMark subtly embeds user-specific information into generated code by controlling the token selection process, ensuring the watermark is imperceptible. Additionally, MCGMark dynamically adjusts the token selection range to induce the LLM to favor high-probability tokens, thus ensuring code quality. Furthermore, by leveraging code structure, MCGMark avoids embedding watermarks into regions easily modified by attackers, such as comments and variable names, enhancing robustness against tampering. Experiments on several advanced LLMs show that MCGMark successfully embeds watermarks in approximately \(85\%\) of cases, under the constraint of a \(400\) -token limit. Moreover, it maintains code quality and demonstrates strong resilience against common code modification. This approach offers a practical solution for tracing malicious code and mitigating the misuse of LLMs.

24 August 2026 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
ZWZexu Wang
Zexu Wang
JCJiachi Chen
Jiachi Chen
YWYanlin Wang
Yanlin Wang
KNKaiwen Ning
Kaiwen Ning
ZLZewei Lin
Zewei Lin
ZZZibin Zheng
Zibin Zheng
With the increasing development of Solidity contracts on Ethereum , more developers are reusing them on other compatible blockchains. However, developers may overlook the differences between the designs of the blockchain system, such as the Gas Mechanism and Consensus Protocol , leading to the same contracts on different blockchains not being able to achieve consistent execution as on Ethereum . This inconsistency reveals design flaws in reused contracts, exposing code smells that hinder code reusability, and we define this inconsistency as EVM-Inequivalent Code Smells . In this paper, we conducted an empirical study to reveal the causes and characteristics of EVM-Inequivalent Code Smells . To ensure the identified smells reflect real developer concerns, we analyzed 1,379 security audit reports, 823 bug bounty reports, and 326 Stack Overflow posts related to reused contracts on EVM-compatible blockchains, such as Binance Smart Chain (BSC) and Polygon . Using the Open Card Sorting method, we defined nine types of EVM-Inequivalent Code Smells . To enable efficient detection, we developed EquivSage , a tool that leverages the contextual understanding capabilities of large language models (LLMs) to guide slicing and static taint analysis via task-specific prompts. Symbolic execution is integrated to improve detection reliability. An analysis of 1,263,683 contracts across six EVM-compatible blockchains using EquivSage reveals that, on average, 15.45% contain at least one EVM-Inequivalent Code Smell , underscoring its widespread prevalence. Since 2024, code smells in reused contracts on Ethereum , Polygon , and Optimism have increased significantly. While not all instances lead to financial loss, high frequency and asset exposure highlight the risks inherent in contract reuse. Developers are encouraged to avoid copy-and-paste practices and to detect such smells proactively before reuse.

12 April 2026
Verified
Conference Paper
Open access
ZWZexu Wang
Zexu Wang
JCJiachi Chen
Jiachi Chen
ZLZewei Lin
Zewei Lin
WCWenqing Chen
Wenqing Chen
KNKaiwen Ning
Kaiwen Ning
ZZZibin Zheng
Zibin Zheng
Smart contracts have significantly advanced blockchain technology, and digital signatures are crucial for reliable verification of contract authority. Through signature verification, smart contracts can ensure that signers possess the required permissions, thus enhancing security and scalability. However, lacking checks on signature usage conditions can lead to repeated verifications, increasing the risk of permission abuse and threatening contract assets. We define this issue as the Signature Replay Vulnerability (SRV). In this paper, we conducted the first empirical study to investigate the causes and characteristics of the SRVs. From 1,419 audit reports across 37 blockchain security companies, we identified 108 with detailed SRV descriptions and classified five types of SRVs. To detect these vulnerabilities automatically, we designed LASiR, which utilizes the general semantic understanding ability of Large Language Models (LLMs) to assist in the static taint analysis of the signature state and identify the signature reuse behavior. It also employs path reachability verification via symbolic execution to ensure effective and reliable detection. To evaluate the performance of LASiR, we conducted large-scale experiments on 15,383 contracts involving signature verification, selected from the initial dataset of 918,964 contracts across four blockchains: Ethereum, Binance Smart Chain, Polygon, and Arbitrum. The results indicate that SRVs are widespread, with affected contracts holding $4.76 million in active assets. Among these, 19.63% of contracts that use signatures on Ethereum contain SRVs. Furthermore, manual verification demonstrates that LASiR achieves an F1-score of 87.90% for detection. Ablation studies and comparative experiments reveal that the semantic information provided by LLMs aids static taint analysis, significantly enhancing LASiR’s detection performance.

30 June 2026 in Proceedings of the ACM on Software Engineering
Verified
Journal Article
Open access
JZJingwen Zhang
Jingwen Zhang
YNYuhong Nan
Yuhong Nan
KNKaiwen Ning
Kaiwen Ning
MYMingxi Ye
Mingxi Ye
WLWei Li
Wei Li
ZZZibin Zheng
Zibin Zheng
Smart contracts are a critical component of blockchain systems. Due to the large amount of digital assets carried by smart contracts, their security is of critical importance. Although numerous tools have been developed for detecting smart contract vulnerability, their effectiveness remains limited, particularly due to the high false positives included in the reported results. Therefore, developers and auditors are often overwhelmed with manually verifying the reported issues. A fundamental reason behind this is that while a reported vulnerability satisfies specific vulnerable patterns, it may not actually be exploitable, either because the vulnerable code cannot be triggered or it does not result in any financial loss. In this paper, we propose V2E, a new framework for validating whether a reported vulnerability is truly exploitable. The core idea of V2E is to automatically generate executable Proof-of-Concept Exploit (PoC for short), and then assess if the vulnerability could be triggered and incur any real damage (i.e., causing financial loss) by the PoC. While LLMs have shown proficiency in PoC generation, achieving our task is by no means trivial. In detail, it is difficult for LLM to: (1) generate and update PoC to trigger a specific vulnerability, (2) evaluate the PoC’s effectiveness to validate exploitable vulnerability. To this end, V2E automates the whole process through a novel combination of PoC generation, validation, and refinement: (1) Firstly, V2E generates targeted PoCs by analyzing potential vulnerability paths. (2) Then, V2E verifies the validity of PoCs through triggerability and profitability analysis. (3) In addition, V2E iteratively refines the generated PoC based on PoC execution feedback, therefore, increasing the chance to confirm the vulnerability. Evaluation on 264 manually labeled contracts shows that V2E outperforms the baseline approach. Particularly, V2E successfully identifies 102 out of 124 exploitable vulnerabilities, achieving a precision of 91.9% and a recall of 82.3%. In addition, it successfully eliminates 71 out of 140 false alarms (50.7%). Besides, V2E effectively enhances the performance of SOTA tools. It reduces the false positive rates of Slither by 76.9%, Mythril by 56.9% and Confuzzius by 65%.