In recent years, cyber attacks have shown diversified, purposeful, and organized characteristics, which pose significant challenges to cyber defense decision-making on internal networks. Due to the continuous confrontation between attackers and defenders, only using data-based statistical or supervised learning methods cannot cope with increasingly severe security threats. It is urgent to rethink network defense from the perspective of decision-making, and prepare for every possible situation. Reinforcement learning has made great breakthroughs in addressing complicated decision-making problems. We propose a framework that defines four modules based on the life cycle of threats: pentest, design, response, recovery. Our aims are to clarify the problem boundary of network defense decision-making problems, to study the problem characteristics in different contexts, to compare the strengths and weaknesses of existing research, and to identify promising challenges for future work. Our work provides a systematic view for understanding and solving decision-making problems in the application of reinforcement learning to cyber defense.

Due to the limited understanding of the physical and chemical processes involved in ozone formation, as well as the large uncertainties surrounding its precursors, commonly used methods often result in biased predictions. Deep learning, as a powerful tool for fitting data, offers an alternative approach. However, most deep learning-based ozone-prediction models only take into account temporality and have limited capacity. Existing spatiotemporal deep learning models generally suffer from model complexity and inadequate spatiality learning. Thus, we propose a novel spatiotemporal model, namely the Spatiotemporal Attentive Gated Recurrent Unit (STAGRU). STAGRU uses a double attention mechanism, which includes temporal and spatial attention layers. It takes historical sequences from a target monitoring station and its neighboring stations as input to capture temporal and spatial information, respectively. This approach enables the achievement of more accurate results. The novel model was evaluated by comparing it to ozone observations in five major cities, Nanjing, Chengdu, Beijing, Guangzhou and Wuhan. All of these cities experience severe ozone pollution. The comparison involved Seq2Seq models, Seq2Seq+Attention models and our models. The experimental results show that our algorithm performs 14% better than Seq2Seq models and 4% better than Seq2Seq+Attention models. We also discuss the interpretability of our method, which reveals that temporality involves short-term dependency and long-term periodicity, while spatiality is mainly reflected in the transportation of ozone with the wind. This study emphasizes the significant impact of transportation on the implementation of ozone-pollution-control measures by the Chinese government.

This paper presents a novel multi-dimensional asymmetric game model for network attack–defense decision-making, called “Catch the Cyber Thief”. The model is built upon the concept of partially observable stochastic games (POSG) and is designed to systematically incorporate multi-dimensional asymmetry into network attack–defense problems. The attack agent is called a “thief” who wants to control a key host by exploring the unfamiliar network environment, and the defense agent is called a “police” who needs to catch the opponent before its goal is accomplished. The results indicate that the asymmetry of network attack and defense is not only influenced by attack and defense strategies but also by spatio-temporal factors such as the attacker’s initial position, network topology, and defense lag time. In addition, we have found that there may exist the “attack rhythm,” which makes “how to maintain a good attack rhythm” and “how to generate a robust defense strategy against different attackers” worth exploring. Compared with existing attack–defense game models, our game model can better generate a direct mapping relationship with real elements, enabling us to understand network attack and defense interactions better, recognize security risks, and design defense strategies that can directly serve real-world decision-making.