Fengqing Jiang, Zhangchen Xu, Luyao Niu, Zhen Xiang, Bhaskar Ramasubramanian, Bo Li, Radha Poovendran. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, Radha Poovendran. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.
Compared with the traditional usage of large language models (LLMs) where users directly send queries to an LLM, LLM-integrated applications serve as middleware to refine users' queries with domain-specific knowledge to better inform LLMs and enhance the responses. However, LLM-integrated applications also introduce new attack surfaces. This work considers a setup where the user and LLM interact via an application in the middle. We focus on the interactions that begin with user's queries and end with LLM-integrated application returning responses to the queries, powered by LLMs at the service backend. We identify potential high-risk vulnerabilities in this setting that can originate from the malicious application developer or from an outsider threat initiator that can control the database access, manipulate and poison high-risk data for the user. Successful exploits of the identified vulnerabilities result in the users receiving responses tailored to the intent of a threat initiator. We assess such threats against LLM-integrated applications empowered by GPT-3.5 and GPT-4. Our experiments show that the threats can effectively bypass the restrictions and moderation policies of OpenAI, resulting in users exposing to the risk of bias, toxic content, privacy, and disinformation. We develop a lightweight, threat-agnostic defense to mitigate insider and outsider threats. Our evaluations demonstrate the efficacy of our defense.
Temporal logic provides a rigorous and expressive way to model specifications for cyber-physical systems (CPS). Such systems, which are playing increasingly important roles in various domains, are also subject to malicious attacks initiated by intelligent adversaries. Malicious attacks may prevent the system from satisfying the desired specification. In this article, we introduce a control synthesis method to minimize deviations from the desired specification due to adversaries. We consider the scenario where the adversary has limited capability of observing the controller's strategy. We synthesize a minimum violation control strategy for a finite-state stochastic game under cosafe linear temporal logic constraints. We formulate a mixed-integer nonlinear program for computing the optimal controller strategy and present two algorithms. First, we propose an exact algorithm that computes the optimal control strategy with probability one but without guarantees on the convergence rate. Second, we present an approximate algorithm that computes a suboptimal control strategy. The approximate algorithm converges faster compared to the exact algorithm; however, it only returns an optimal solution when the adversary has full observation over the controller's strategy. A numerical case study is presented to evaluate the proposed framework.