28 October 2025 in IEEE Transactions on Information Forensics and Security
With the enhanced performance of large language models (LLMs) on natural language processing tasks, potential moral and ethical issues of LLMs arise in the Web service. Malicious attackers exist who induce LLMs to jailbreak and generate information containing illegal, privacy-invasive information through techniques such as prompt engineering. As a result, LLMs counter malicious attackers’ attacks using techniques such as safety alignment. However, the strong defense mechanism of the LLMs through rejection replies, which are typically unrelated to the attacker’s query, is easily identified by attackers and leaks information that strengthens their capabilities. We propose an LLM-driven defense task that generates safe and disguised responses which remain relevant to the attack request while concealing defensive intent and withholding any useful information from the attacker. To achieve this task, we construct an LLM-based multi-agent attacker-disguiser game that trains two LLM-based players to select in-context learning (ICL) examples and strategies, guiding them to better generation and approach to Nash equilibrium. To enhance the LLMs’ generation via ICL, we apply the Minimax Q-learning to optimize the players for choosing ICL examples. To further refine generation, we use policy gradient (PG) to train the strategy selectors to select prompt strategies for LLMs. To improve the ICL examples and prompts for LLMs’ generation, we iteratively train the players with ICL sample selection training and prompt strategy training. The goal of this game is to arrive at the Nash equilibrium. Through training, we obtain an LLM-based defender which can receive jailbreak prompts from malicious users and output safe and disguised responses. The experiments reveal that our method strengthens the LLM’s ability to disguise the defense intent compared to other methods. Our method also minimizes the attacker-perceived drop in attack success rate, while delivering the largest actual reduction. This effectively misleads the attacker’s evaluation and enhances practical defense utility.