Xiaoyu Zhang

Xiaoyu Zhang

Auto-generated profile, based on publicly available information

Xiaoyu Zhang

Nanyang technological university, Singapore, Singapore

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
11 December 2025 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
XZXiaoyu Zhang
Xiaoyu Zhang
CZCen Zhang
Cen Zhang
TLTianlin Li
Tianlin Li
YHYihao Huang
Yihao Huang
XJXiaojun Jia
Xiaojun Jia
SCShen Chao
Shen Chao
The systems and software powered by Large Language Models (LLMs) and Multi-Modal Large Language Models (MLLMs) have played a critical role in numerous scenarios. However, current LLM systems are vulnerable to prompt-based attacks, with jailbreaking attacks enabling the LLM system to generate harmful content, while hijacking attacks manipulate the LLM system to perform attacker-desired tasks, underscoring the necessity for detection tools. Unfortunately, existing detecting approaches are usually tailored to specific attacks, resulting in poor generalization in detecting various attacks across different modalities. To address it, we propose JailGuard , a universal detection framework deployed on top of LLM systems for prompt-based attacks across text and image modalities. JailGuard operates on the principle that attacks are inherently less robust than benign ones. Specifically, JailGuard mutates untrusted inputs to generate variants and leverages the discrepancy of the variants’ responses on the target model to distinguish attack samples from benign samples. We implement 18 mutators for text and image inputs and design a mutator combination policy to further improve detection generalization. The evaluation on the dataset containing 15 known attack types suggests that JailGuard achieves the best detection accuracy of 86.14%/82.90% on text and image inputs, outperforming state-of-the-art methods by 11.81–25.73% and 12.20–21.40%.

24 November 2023 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
YZYuhan Zhi
Yuhan Zhi
XXXiaofei Xie
Xiaofei Xie
SCShen Chao
Shen Chao
JSJun Sun
Jun Sun
XZXiaoyu Zhang
Xiaoyu Zhang
XGXiaohong Guan
Xiaohong Guan
Deep learning (DL) has been applied in many applications. Meanwhile, the quality of DL systems is becoming a big concern. To evaluate the quality of DL systems, a number of DL testing techniques have been proposed. To generate test cases, a set of initial seed inputs are required. Existing testing techniques usually construct seed corpus by randomly selecting inputs from training or test dataset. Till now, there is no study on how initial seed inputs affect the performance of DL testing and how to construct an optimal one. To fill this gap, we conduct the first systematic study to evaluate the impact of seed selection strategies on DL testing. Specifically, considering three popular goals of DL testing (i.e., coverage, failure detection, and robustness), we develop five seed selection strategies, including three based on single-objective optimization (SOO) and two based on multi-objective optimization (MOO). We evaluate these strategies on seven testing tools. Our results demonstrate that the selection of initial seed inputs greatly affects the testing performance. SOO-based selection can construct the best seed corpus that can boost DL testing with respect to the specific testing goal. MOO-based selection strategies can construct seed corpus that achieve balanced improvement on multiple objectives.

5 March 2025 in ACM Computing Surveys
Verified
Journal Article
Open access
XZXiaoyu Zhang
Xiaoyu Zhang
WJWeipeng Jiang
Weipeng Jiang
SCShen Chao
Shen Chao
QLQi Li
Qi Li
QWQian Wang
Qian Wang
XGXiaohong Guan
Xiaohong Guan
Recently, software systems powered by deep learning (DL) techniques have significantly facilitated people’s lives in many aspects. As the backbone of these DL systems, various DL libraries undertake the underlying optimization and computation. However, like traditional software, DL libraries are not immune to bugs. These bugs may be propagated to programs and software developed based on DL libraries, thereby posing serious threats to users’ personal property and safety. Studying the characteristics of DL libraries, their associated bugs, and the corresponding testing methods is crucial for enhancing the security of DL systems and advancing the widespread application of DL technology. This paper provides an overview of the testing research on various DL libraries, discusses the strengths and weaknesses of existing methods, and provides guidance and reference for the application of DL library testing methods. This paper first introduces the workflow of DL underlying libraries and the characteristics of three kinds of DL libraries involved, namely DL framework, DL compiler, and DL hardware library. Subsequently, this paper constructs a literature collection pipeline and comprehensively summarizes existing testing methods on these DL libraries to analyze their effectiveness and limitations. It also reports findings and the challenges of existing DL library testing in real-world applications for future research.

26 April 2025 in ACM Transactions on Software Engineering and Methodology
Verified
Journal Article
Open access
XZXiaoyu Zhang
Xiaoyu Zhang
JZJuan Zhai
Juan Zhai
SMShiqing Ma
Shiqing Ma
XGXiaohong Guan
Xiaohong Guan
SCShen Chao
Shen Chao
Deep Learning models have become an integrated component of modern software systems. In response to the challenge of model design, researchers proposed Automated Machine Learning (AutoML) systems, which automatically search for model architecture and hyperparameters for a given task. Like other software systems, existing AutoML systems have shortcomings in their design. We identify two common and severe shortcomings in AutoML, performance issue (i.e., searching for the desired model takes an unreasonably long time) and ineffective search issue (i.e., AutoML systems are not able to find an accurate enough model). After analyzing the workflow of AutoML, we observe that existing AutoML systems overlook potential opportunities in search space, search method, and search feedback, which results in performance and ineffective search issues. Based on our analysis, we design and implement DREAM , an automatic and general-purpose tool to alleviate and repair the shortcomings of AutoML pipelines and conduct effective model searches for diverse tasks. It monitors the process of AutoML to collect detailed feedback and automatically repairs shortcomings by expanding search space and leveraging a feedback-driven search strategy. Our evaluation results show that DREAM can be applied on two state-of-the-art AutoML pipelines and effectively and efficiently repair their shortcomings.

12 July 2024 in Proceedings of the ACM on Software Engineering
Verified
Journal Article
Open access
WJWeipeng Jiang
Weipeng Jiang
JZJuan Zhai
Juan Zhai
SMShiqing Ma
Shiqing Ma
XZXiaoyu Zhang
Xiaoyu Zhang
SCShen Chao
Shen Chao
Large language models have gained significant popularity and are often provided as a service (i.e., LLMaaS). Companies like OpenAI and Google provide online APIs of LLMs to allow downstream users to create innovative applications. Despite its popularity, LLM safety and quality assurance is a well-recognized concern in the real world, requiring extra efforts for testing these LLMs. Unfortunately, while end-to-end services like ChatGPT have garnered rising attention in terms of testing, the LLMaaS embeddings have comparatively received less scrutiny. We state the importance of testing and uncovering problematic individual embeddings without considering downstream applications. The abstraction and non-interpretability of embedded vectors, combined with the black-box inaccessibility of LLMaaS, make testing a challenging puzzle. This paper proposes C ostello , a black-box approach to reveal potential defects in abstract embedding vectors from LLMaaS by contrastive testing . Our intuition is that high-quality LLMs can adequately capture the semantic relationships of the input texts and properly represent their relationships in the high-dimensional space. For the given interface of LLMaaS and seed inputs, Costello can automatically generate test suites and output words with potential problematic embeddings. The idea is to synthesize contrastive samples with guidance, including positive and negative samples, by mutating seed inputs. Our synthesis guide will leverage task-specific properties to control the mutation procedure and generate samples with known partial relationships in the high-dimensional space. Thus, we can compare the expected relationship (oracle) and embedding distance (output of LLMs) to locate potential buggy cases. We evaluate C ostello on 42 open-source (encoder-based) language models and two real-world commercial LLMaaS. Experimental results show that C ostello can effectively detect semantic violations, where more than 62% of violations on average result in erroneous behaviors (e.g., unfairness) of downstream applications.

5 July 2026
Verified
Conference Paper
Open access
XZXinxue Zhu
Xinxue Zhu
JWJiacong Wu
Jiacong Wu
XZXiaoyu Zhang
Xiaoyu Zhang
TLTianlin Li
Tianlin Li
YMYanzhou Mu
Yanzhou Mu
YLYang Liu
Yang Liu