Jinyuan Jia

Jinyuan Jia

Auto-generated profile, based on publicly available information

Jinyuan Jia

Pennsylvania State University

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 January 2024
Verified
Conference Paper
Open access
PDF file
ZXZhangchen Xu
Zhangchen Xu
FJFengqing Jiang
Fengqing Jiang
LNLuyao Niu
Luyao Niu
JJJinyuan Jia
Jinyuan Jia
BLBill Yuchen Lin
Bill Yuchen Lin
RPRadha Poovendran
Radha Poovendran
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, Radha Poovendran. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

1 January 2025
Verified
Conference Paper
Open access
PDF file
ZWZixuan Weng
Zixuan Weng
XJXiaolong Jin
Xiaolong Jin
JJJinyuan Jia
Jinyuan Jia
XZXiangyu Zhang
Xiangyu Zhang
Zixuan Weng, Xiaolong Jin, Jinyuan Jia, Xiangyu Zhang. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.

1 January 2024
Verified
Conference Paper
Open access
PDF file
HZHangfan Zhang
Hangfan Zhang
ZGZhimeng Guo
Zhimeng Guo
HZHuaisheng Zhu
Huaisheng Zhu
BCBochuan Cao
Bochuan Cao
LLLu Lin
Lu Lin
DWDinghao Wu
Dinghao Wu
Hangfan Zhang, Zhimeng Guo, Huaisheng Zhu, Bochuan Cao, Lu Lin, Jinyuan Jia, Jinghui Chen, Dinghao Wu. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

11 April 2025 in Proceedings of the AAAI Conference on Artificial Intelligence
Verified
Journal Article
Open access
PDF file
YLYupei Liu
Yupei Liu
YWYanting Wang
Yanting Wang
JJJinyuan Jia
Jinyuan Jia
An image encoder pre-trained by self-supervised learning can be used as a general-purpose feature extractor to build downstream classifiers for various downstream tasks. However, many studies showed that an attacker can embed a trojan into an encoder such that multiple downstream classifiers built based on the trojaned encoder simultaneously inherit the trojan behavior. In this work, we propose TrojanDec, the first data-free method to identify and recover a test input embedded with a trigger. Given a (trojaned or clean) encoder and a test input, TrojanDec first predicts whether the test input is trojaned. If not, the test input is processed in a normal way to maintain the utility. Otherwise, the test input will be further restored to remove the trigger. Our extensive evaluation shows that TrojanDec can effectively identify the trojan (if any) from a given test input and recover it under state-of-the-art trojan attacks. We further demonstrate by experiments that our TrojanDec outperforms the state-of-the-art defenses.

29 November 2023 in arXiv
Verified
Preprint
Open access
PDF file
FJFengqing Jiang
Fengqing Jiang
ZXZhangchen Xu
Zhangchen Xu
LNLuyao Niu
Luyao Niu
BWBoxin Wang
Boxin Wang
JJJinyuan Jia
Jinyuan Jia
RPRadha Poovendran
Radha Poovendran
Large language models (LLMs) are increasingly deployed as the service backend for LLM-integrated applications such as code completion and AI-powered search. LLM-integrated applications serve as middleware to refine users' queries with domain-specific knowledge to better inform LLMs and enhance the responses. Despite numerous opportunities and benefits, LLM-integrated applications also introduce new attack surfaces. Understanding, minimizing, and eliminating these emerging attack surfaces is a new area of research. In this work, we consider a setup where the user and LLM interact via an LLM-integrated application in the middle. We focus on the communication rounds that begin with user's queries and end with LLM-integrated application returning responses to the queries, powered by LLMs at the service backend. For this query-response protocol, we identify potential vulnerabilities that can originate from the malicious application developer or from an outsider threat initiator that is able to control the database access, manipulate and poison data that are high-risk for the user. Successful exploits of the identified vulnerabilities result in the users receiving responses tailored to the intent of a threat initiator. We assess such threats against LLM-integrated applications empowered by OpenAI GPT-3.5 and GPT-4. Our empirical results show that the threats can effectively bypass the restrictions and moderation policies of OpenAI, resulting in users receiving responses that contain bias, toxic content, privacy risk, and disinformation. To mitigate those threats, we identify and define four key properties, namely integrity, source identification, attack detectability, and utility preservation, that need to be satisfied by a safe LLM-integrated application. Based on these properties, we develop a lightweight, threat-agnostic defense that mitigates both insider and outsider threats.

23 May 2025 in arXiv
Verified
Preprint
Open access
PDF file
YJYuqi Jia
Yuqi Jia
ZSZedian Shao
Zedian Shao
YLYupei Liu
Yupei Liu
JJJinyuan Jia
Jinyuan Jia
DSDawn Song
Dawn Song
NGNeil Zhenqiang Gong
Neil Zhenqiang Gong
Large Language Models (LLMs) are vulnerable to prompt injection attacks, and several defenses have recently been proposed, often claiming to mitigate these attacks successfully. However, we argue that existing studies lack a principled approach to evaluating these defenses. In this paper, we argue the need to assess defenses across two critical dimensions: (1) effectiveness, measured against both existing and adaptive prompt injection attacks involving diverse target and injected prompts, and (2) general-purpose utility, ensuring that the defense does not compromise the foundational capabilities of the LLM. Our critical evaluation reveals that prior studies have not followed such a comprehensive evaluation methodology. When assessed using this principled approach, we show that existing defenses are not as successful as previously reported. This work provides a foundation for evaluating future defenses and guiding their development. Our code and data are available at: https://github.com/PIEval123/PIEval.