Reinforcement learning has shown a great ability and has defeated human beings in the field of real-time strategy games. In recent years, reinforcement learning has been used in cyberspace to carry out automated and intelligent attacks. Traditional defense methods are not enough to deal with this problem, so it is necessary to design defense agents to counter intelligent attacks. The interaction between the attack agent and the defense agent can be modeled as a multi-agent Markov game. In this paper, an adversarial decision-making approach that combines the Bayesian Strong Stackelberg and the WoLF algorithms was proposed to obtain the equilibrium point of multi-agent Markov games. With this method, the defense agent can obtain the adversarial decision-making strategy as well as continuously adjust the strategy in cyberspace. As verified in experiments, the defense agent should attach importance to short-term rewards in the process of a real-time game between the attack agent and the defense agent. The proposed approach can obtain the largest rewards for defense agent compared with the classic Nash-Q and URS-Q algorithms. In addition, the proposed approach adjusts the action selection probability dynamically, so that the decision entropy of optimal action gradually decreases.

The penetration test has many repetitive operations and requires advanced expert knowledge, therefore, the manual penetration test is inefficient. With the development of reinforcement learning, the intelligent penetration test has been a research hotspot. However, the existing intelligent penetration test simulation environments only focus on the exploits of target hosts by the penetration tester agent’s actions while ignoring the important role of social engineering in the penetration test in reality. In addition, the construction of the existing penetration test simulation environment is based on the traditional network graph model without integrating security factors and attributes, and it is difficult to express the interaction between the penetration tester and the target network. This paper constructs an improved network graph model for penetration test (NMPT), which integrates the relevant security attributes of the penetration test. The NMPT model lays the foundation for extending the penetration tester’s social engineering actions. Then, we propose an intelligent penetration test method that incorporates social engineering factors (SE-AIPT) based on the Markov Decision Process. We adopt several mainstream reinforcement learning algorithms to train attack agents. The experiments show that the SE-AIPT method could vividly model the penetration tester agent’s social engineering actions, which effectively improves the reality of the simulation environment. Moreover, the penetration tester agent shows superior effects in the attack path discovery in the intelligent penetration test simulation environment constructed by the SE-AIPT method.

In this paper, in order to accurately detect Domain Name System (DNS) covert channels based on DNS over HTTPS (DoH) encryption and to solve the problems of weak single-feature differentiation and poor performance in the existing detection methods, we have designed a DoH-encrypted DNS covert channel detection method based on features fusion, called FF-MR. FF-MR is based on a Multi-Head Attention and Residual Neural Network. It fuses session statistical features with multi-channel session byte sequence features. Some important features that play a key role in the detection task are screened out of the fused features through the calculation of the Multi-Head Attention mechanism. Finally, a Multi-Layer Perceptron (MLP) is used to detect encrypted DNS covert channels. By considering both global and focused features, the main idea of FF-MR is that the degree of correlation between each feature and all other features is expressed as an attention weight. Thus, features are re-represented as the result of the weighted fusion of all features using the Multi-Head Attention mechanism. Focusing on certain important features according to the distribution of attention weights improves the detection performance. While detecting the traffic in encrypted DNS covert channels, FF-MR can also accurately identify encrypted traffic generated by the three DNS covert channel tools. Experiments on the CIRA-CIC-DoHBrw-2020 dataset show that the macro-averaging recall and precision of the FF-MR method reach 99.73% and 99.72%, respectively, and the macro-averaging F1-Score reached 0.9978, which is up to 4.56% higher than the existing methods compared in the paper. FF-MR achieves at most an 11.32% improvement in macro-averaging F1-Score in identifying three encrypted DNS covert channels, indicating that FF-MR has a strong ability to detect and identify DoH-encrypted DNS covert channels.