Reinforcement learning has shown a great ability and has defeated human beings in the field of real-time strategy games. In recent years, reinforcement learning has been used in cyberspace to carry out automated and intelligent attacks. Traditional defense methods are not enough to deal with this problem, so it is necessary to design defense agents to counter intelligent attacks. The interaction between the attack agent and the defense agent can be modeled as a multi-agent Markov game. In this paper, an adversarial decision-making approach that combines the Bayesian Strong Stackelberg and the WoLF algorithms was proposed to obtain the equilibrium point of multi-agent Markov games. With this method, the defense agent can obtain the adversarial decision-making strategy as well as continuously adjust the strategy in cyberspace. As verified in experiments, the defense agent should attach importance to short-term rewards in the process of a real-time game between the attack agent and the defense agent. The proposed approach can obtain the largest rewards for defense agent compared with the classic Nash-Q and URS-Q algorithms. In addition, the proposed approach adjusts the action selection probability dynamically, so that the decision entropy of optimal action gradually decreases.

The penetration test has many repetitive operations and requires advanced expert knowledge, therefore, the manual penetration test is inefficient. With the development of reinforcement learning, the intelligent penetration test has been a research hotspot. However, the existing intelligent penetration test simulation environments only focus on the exploits of target hosts by the penetration tester agent’s actions while ignoring the important role of social engineering in the penetration test in reality. In addition, the construction of the existing penetration test simulation environment is based on the traditional network graph model without integrating security factors and attributes, and it is difficult to express the interaction between the penetration tester and the target network. This paper constructs an improved network graph model for penetration test (NMPT), which integrates the relevant security attributes of the penetration test. The NMPT model lays the foundation for extending the penetration tester’s social engineering actions. Then, we propose an intelligent penetration test method that incorporates social engineering factors (SE-AIPT) based on the Markov Decision Process. We adopt several mainstream reinforcement learning algorithms to train attack agents. The experiments show that the SE-AIPT method could vividly model the penetration tester agent’s social engineering actions, which effectively improves the reality of the simulation environment. Moreover, the penetration tester agent shows superior effects in the attack path discovery in the intelligent penetration test simulation environment constructed by the SE-AIPT method.

Increasingly, cyber security personnel are using cyber deception defense techniques to deal with network intrusions. However, traditional cyber deception techniques (such as honeypots and honeynets) are easily detected by attackers, thus leading to failure. Therefore, we propose a cyber deception defense method based on the signal game to improve the effectiveness of the defense. More specifically, first, we propose a moving target defense (MTD) enhanced cyber deception defense mechanism. Then, on the basis of the in-depth analysis of network attack and defense scenarios, a signal game model is constructed to describe the network attack and defense process, and a multistage attack and defense game equilibrium solution method is designed to guide the selection of the optimal deception defense strategy. Meanwhile, considering the actual network attack and defense, we quantify the game revenue based on a probabilistic model. The experimental results show that the defense method proposed in this paper could guide the defender to implement the optimal defense strategy and achieve a better defense effect.

In this paper, in order to accurately detect Domain Name System (DNS) covert channels based on DNS over HTTPS (DoH) encryption and to solve the problems of weak single-feature differentiation and poor performance in the existing detection methods, we have designed a DoH-encrypted DNS covert channel detection method based on features fusion, called FF-MR. FF-MR is based on a Multi-Head Attention and Residual Neural Network. It fuses session statistical features with multi-channel session byte sequence features. Some important features that play a key role in the detection task are screened out of the fused features through the calculation of the Multi-Head Attention mechanism. Finally, a Multi-Layer Perceptron (MLP) is used to detect encrypted DNS covert channels. By considering both global and focused features, the main idea of FF-MR is that the degree of correlation between each feature and all other features is expressed as an attention weight. Thus, features are re-represented as the result of the weighted fusion of all features using the Multi-Head Attention mechanism. Focusing on certain important features according to the distribution of attention weights improves the detection performance. While detecting the traffic in encrypted DNS covert channels, FF-MR can also accurately identify encrypted traffic generated by the three DNS covert channel tools. Experiments on the CIRA-CIC-DoHBrw-2020 dataset show that the macro-averaging recall and precision of the FF-MR method reach 99.73% and 99.72%, respectively, and the macro-averaging F1-Score reached 0.9978, which is up to 4.56% higher than the existing methods compared in the paper. FF-MR achieves at most an 11.32% improvement in macro-averaging F1-Score in identifying three encrypted DNS covert channels, indicating that FF-MR has a strong ability to detect and identify DoH-encrypted DNS covert channels.

Faced with challenges posed by sophisticated cyber attacks and dynamic characteristics of cyberspace, the autonomous cyber defense (ACD) technology has shown its effectiveness. However, traditional decision-making methods for ACD are unable to effectively characterize the network topology and internode dependencies, which makes it difficult for defenders to identify key nodes and critical attack paths. Therefore, this paper proposes an enhanced decision-making method combining graph embedding with reinforcement learning algorithms. By constructing a game model for cyber confrontations, this paper models important elements of the network topology for decision-making, which guide the defender to dynamically optimize its strategy based on topology awareness. We improve the reinforcement learning with the Node2vec algorithm to characterize information for the defender from the network. And, node attributes and network structural features are embedded into low-dimensional vectors instead of using traditional one-hot encoding, which can address the perceptual bottleneck in high-dimensional sparse environments. Meanwhile, the algorithm training environment Cyberwheel is extended by adding new fine-grained defense mechanisms to enhance the utility and portability of ACD. In experiments, our decision-making method based on graph embedding is compared and analyzed with traditional perception methods. The results show and verify the superior performance of our approach in the strategy selection of defensive decision-making. Also, diverse parameters of the graph representation model Node2vec are analyzed and compared to find the impact on the enhancement of the embedding effectiveness for the decision-making of ACD.