Penetration testing is an effective way to test and evaluate cybersecurity by simulating a cyberattack. However, the traditional methods deeply rely on domain expert knowledge, which requires prohibitive labor and time costs. Autonomous penetration testing is a more efficient and intelligent way to solve this problem. In this paper, we model penetration testing as a Markov decision process problem and use reinforcement learning technology for autonomous penetration testing in large scale networks. We propose an improved deep Q-network (DQN) named NDSPI-DQN to address the sparse reward problem and large action space problem in large-scale scenarios. First, we reasonably integrate five extensions to DQN, including noisy nets, soft Q-learning, dueling architectures, prioritized experience replay, and intrinsic curiosity model to improve the exploration efficiency. Second, we decouple the action and split the estimators of the neural network to calculate two elements of action separately, so as to decrease the action space. Finally, the performance of algorithms is investigated in a range of scenarios. The experiment results demonstrate that our methods have better convergence and scaling performance.
With the development of technology, people’s daily lives are closely related to networks. The importance of cybersecurity protection draws global attention. Automated penetration testing is the novel method to protect the security of networks, which enhances efficiency and reduces costs compared with traditional manual penetration testing. Previous studies have provided many ways to obtain a better policy for penetration testing paths, but many studies are based on ideal penetration testing scenarios. In order to find potential vulnerabilities from the perspective of hackers in the real world, this paper models the process of black-box penetration testing as a Partially Observed Markov Decision Process (POMDP). In addition, we propose a new algorithm named , which is applied to the automated black-box penetration testing. In the POMDP model, an agent interacts with a network environment to choose a better policy without insider information about the target network, except for the start points. To handle this problem, we utilize a Long Short-Term Memory (LSTM) structure empowering agent to make decisions based on historical memory. In addition, this paper enhances the current algorithm using the structure of the neural network, the calculation method of the Q-value, and adding noise parameters to the neural network to advance the generalization and efficiency of this algorithm. In the last section, we conduct comparison experiments of the algorithm and other recent state-of-the-art (SOTA) algorithms. The experimental results vividly show that this novel algorithm is able to find a greater attack-path strategy for all vulnerable hosts in the automated black-box penetration testing. Additionally, the generalization and robustness of this algorithm are far superior to other SOTA algorithms in different size simulation scenarios based on the CyberBattleSim simulation developed by Microsoft.
Penetration testing is an effective method of making computers secure. When conducting penetration testing, it is necessary to fully understand the various elements in the cyberspace. Prediction of future cyberspace state through perception and understanding of cyberspace can assist defenders in decision-making and action execution. Accurate cyberspace detection information is the key to ensuring successful penetration testing. However, cyberspace situation awareness still faces the following challenges. Due to the limited detection capability, the information obtained from cyberspace detection intelligence is incomplete. There are some errors in the cyberspace detection intelligence, which may mislead the penetration testing workers. The knowledge graph can store and manage the cybersecurity data. In order to ensure the integrity and accuracy of cyberspace information, we design a knowledge graph completion model called CSNT to complete cybersecurity data. CSNT uses the BiLSTM to capture the interaction information between entities and relationships. It models the relationship between entities by combining the neural network and tensor decomposition. The Pearson Mix Net is designed to control the generation of joint vectors. We also design a novel self-distillation strategy to reduce catastrophic forgetting during model training. After learning the relationship pattern between entities in the cyberspace detection intelligence, the model can be used to mine the knowledge not found in the cybersecurity detection intelligence and correct the erroneous records. Experiments show that our method has certain advantages for the knowledge graph completion.
Large language models’ domain-specific capabilities can be enhanced through specialized datasets, yet constructing comprehensive cybersecurity datasets remains challenging due to the field’s multidisciplinary nature. We present PenQA, a novel instructional dataset for penetration testing that integrates theoretical and practical knowledge. Leveraging authoritative sources like MITRE ATT&CK™ and Metasploit, we employ online large language models to generate approximately 50,000 question–answer pairs.We demonstrate PenQA’s efficacy by fine-tuning language models with fewer than 10 billion parameters. Evaluation metrics, including the BLEU, ROUGE, and BERTScore, show significant improvements in the models’ penetration testing capabilities. PenQA is designed to be compatible with various model architectures and updatable as new techniques emerge. This work has implications for automated penetration testing tools, cybersecurity education, and decision support systems. The PenQA dataset is available in our GitHub repository.
Web application firewalls (WAFs) are critical defenses against persistent threats to web applications, yet their security evaluation remains challenging. Traditional manual testing methods are often inefficient and resource-intensive, while existing reinforcement learning (RL)-based automated approaches face two key limitations: (1) attackers cannot perceive opaque WAF rule logic; (2) boolean feedback from WAFs results in sparse/delayed rewards—sparse rewards trap agents in blind exploration, and delayed rewards hinder the association between early actions and final outcomes, adversely affecting learning efficiency. To address those challenges, we propose Ouroboros—a framework integrating genetic algorithm-based symbolic rule reconstruction (translating WAF rules into interpretable RNNs for fine-grained confidence scoring), timing side-channel analysis (evaluating rule-matching depth), and a multi-tiered reward mechanism to enable self-evolving RL testing. Experiments show that the framework reaches 89.2% bypass success rate on signature-based WAFs. This paper presents an efficient solution for automated WAF testing and delivers insights for optimizing rule logic and anomaly detection mechanisms.