5G technologies provide ubiquitous connectivity. However, 5G security is a particularly important issue. Moreover, because public datasets are outdated, we need to create a self-generated dataset on the virtual platform. Therefore, we propose a two-stage intelligent detection model to enable 5G networks to withstand security issues and threats. Finally, we define malicious traffic detection capability metrics. We apply the self-generated dataset and metrics to thoroughly evaluate the proposed mechanism. We compare our proposed method with benchmark statistics and neural network algorithms. The experimental results show that the two-stage intelligent detection model can distinguish between benign and abnormal traffic and classify 21 kinds of DDoS. Our analysis also shows that the proposed approach outperforms all the compared approaches in terms of detection rate, malicious traffic detection capability, and response time.

In view of various security requirements, there are various security services in the network. In particular, DDoS attacks have various types and detection methods. How to flexibly combine security services and make full use of the information provided by security services have become urgent problems to be solved. This paper combines the reasoning ability of the malicious behavior knowledge base to realize the dynamic deployment of the service function chain and dynamic configuration of the security service function. The method feeds back the information generated by the security service to the knowledge base. After the analysis of the knowledge base, the service function chain path and the security service configuration policies are generated, and these policies will be dynamically distributed to the security service function. Finally, security services can be dynamically arranged for different network traffic, realizing the coordinated use of various security services and improving the overall detection rate of the network. The experimental results show that by arranging the paths under the UDP and the TCP, the overall detection rate of the network can reach 99% and 88%, respectively, indicating that it has a good overall detection performance for multiple distributed denial of service (DDoS) attacks.

Security service function chaining (SFC) based on software-defined networking (SDN) and network function virtualization (NFV) technology allows traffic to be forwarded sequentially among different security service functions to achieve a combination of security functions. Security SFC can be deployed according to requirements, but the current SFC is not flexible enough and lacks an effective feedback mechanism. The SFC is not traffic aware and the changes of traffic may cause the previously deployed security SFC to be invalid. How to establish a closed-loop mechanism to enhance the adaptive capability of the security SFC to malicious traffic has become an important issue. Our contribution is threefold. First, we propose a secure SFC path selection framework. The framework can accept the feedback results of traffic and security service functions in SFC, and dynamically select the optimal path for SFC based on the feedback results. It also realizes the automatic deployment of paths, forming a complete closed loop. Second, we expand the protocol of SFC to realize the security SFC with branching path, which improve flexibility of security SFC. Third, we propose a deep reinforcement learning-based dynamic path selection method for security SFC. It infers the optimal branching path by analyzing feedback from the security SFC. We have experimented with Distributed Denial of Service (DDoS) attack detection modules as security service functions. Experimental results show that our proposed method can dynamically select the optimal branching path for a security SFC based on traffic features and the state of the SFC. And it improves the accuracy of the overall malicious traffic detection of the security SFC and significantly reduces the latency and overall load of the SFC.

5G suffers from new security and privacy issues. More specifically, distributed denial-of-service (DDoS) is one of the most common and destructive attack types. The traditional network does not perform well, because they do not quickly support the deployment of new service function as well as provide network security service on demand. Recently, with the rise of Reinforcement Learning (RL), Software Defined Network (SDN)/Network Functions Virtualization (NFV), and Interface to Network Security Functions (I2NSF) techniques, they play a significant role in security enhancement. Hence, we propose a framework to integrate RL with SDN/NFV and I2NSF architecture (RL-SDNV), in which the RL agent can intelligently select suitable security function chain (SFC) under attack scenarios. Then, we model the DDoS detection problem as a Markov Decision Process (MDP), and a QLearning detection algorithm (QLSFC) is proposed, in which the designed reward includes the processing time and the malicious traffic reduction rate. Finally, we build a corresponding prototype system and verify the feasibility and effectiveness of the proposed algorithm through compared experiments with other RL algorithms.