Cybersecurity is a growing concern in today’s interconnected world. Traditional cybersecurity approaches, such as signature-based detection and rule-based firewalls, are often limited in their ability to effectively respond to evolving and sophisticated cyber threats. Reinforcement learning (RL) has shown great potential in solving complex decision-making problems in various domains, including cybersecurity. However, there are significant challenges to overcome, such as the lack of sufficient training data and the difficulty of modeling complex and dynamic attack scenarios hindering researchers’ ability to address real-world challenges and advance the state of the art in RL cyber applications. In this work, we applied a deep RL (DRL) framework in adversarial cyber-attack simulation to enhance cybersecurity. Our framework uses an agent-based model to continuously learn from and adapt to the dynamic and uncertain environment of network security. The agent decides on the optimal attack actions to take based on the state of the network and the rewards it receives for its decisions. Our experiments on synthetic network security show that the DRL approach outperforms existing methods in terms of learning optimal attack actions. Our framework represents a promising step towards the development of more effective and dynamic cybersecurity solutions.

Recently, the IT industry has become larger, and cloud service has rapidly increased; thus cybersecurity to protect sensitive data from attacks has become an important factor. However, cloud services have become larger, making the surface area larger, and a complex cyber environment leads to difficulty managing and defending. With the rise of artificial intelligence, applying artificial intelligence to a cyber environment to automatically detect and respond to cyberattacks has begun to get attention. In order to apply artificial intelligence in cyber environments, a simulation framework that is easily applicable and can represent real situations well is needed. In this study, we introduce the framework Cyber Environment (CYE) that provides useful components that abstract complex and large cloud environments. Additionally, we use CYE to reproduce real-world situations into the scenario and apply reinforcement learning for training automated intelligence defense agents.