Today, with the continuous promotion and development of IoT and 5G technology, Cyberspace has become an important pillar of economic and social development, and also a foundational domain of national security. Cyberspace security is attracting more and more attention. Therefore, detecting malware and its variants is of great significance to Cyberspace. However, the increasing sophistication of malicious variants, such as encryption, polymorphism and obfuscation, makes it more difficult to identified malware effectively. In this paper, a malware detection method of code texture visualization based on an improved Faster RCNN (Region-Convolutional Neural Networks) combining transfer learning is proposed. We utilize visualization technology to map malicious code into corresponding images with typical texture features, and realize the classification of malware. Firstly, in order to quickly acquire and locate the representative texture of malware, we adopt CNN to extract the global and deeper features of malicious code images. Then with RPN (Region Proposal Network) we generate the target image frame, which is used to locate the core texture of malware file (.text file), to realize the accurate positioning of malicious features. Secondly, we preprocess and train Faster RCNN model with ImageNet set, and then transfer the model to the malware classification model to accelerate the convergence of the first model and promote generation performance. Thirdly, we construct an improved objective function in which a novel multi-label of classification proportion is added to solve the problem that the texture change of “.text” section and other sections in malicious code image is not obvious after transfer learning. We collect code samples of six malware families from Kaggle platform, and compared the experimental results before and after transfer. The results show that the novel method can accelerate the convergence of loss function, and obtain higher accuracy (92.8%), lower FPR (6.8%) and better P-R (precision-recall) curve.

The increasing sophistication of malware variants such as encryption, polymorphism, and obfuscation calls for the new detection and classification technology. In this paper, MalDeep, a novel malware classification framework of deep learning based on texture visualization, is proposed against malicious variants. Through code mapping, texture partitioning, and texture extracting, we can study malware classification in a new feature space of image texture representation without decryption and disassembly. Furthermore, we built a malware classifier on convolutional neural network with two convolutional layers, two downsampling layers, and many full connection layers. We adopt the dataset, from Microsoft Malware Classification Challenge including 9 categories of malware families and 10868 variant samples, to train the model. The experiment results show that the established MalDeep has a higher accuracy rate for malware classification. In particular, for some backdoor families, the classification accuracy of the model reaches over 99%. Moreover, compared with other main antivirus software, MalDeep also outperforms others in the average accuracy for the variants from different families.

With explosive growth of malware, Internet users face enormous threats from Cyberspace, known as “fifth dimensional space.” Meanwhile, the continuous sophisticated metamorphism of malware such as polymorphism and obfuscation makes it more difficult to detect malicious behavior. In the paper, based on the dynamic feature analysis of malware, a novel feature extraction method of hybrid gram (H-gram) with cross entropy of continuous overlapping subsequences is proposed, which implements semantic segmentation of a sequence of API calls or instructions. The experimental results show the H-gram method can distinguish malicious behaviors and is more effective than the fixed-length n-gram in all four performance indexes of the classification algorithms such as ID3, Random Forest, AdboostM1, and Bagging.

The application-layer distributed denial of service (AL-DDoS) attack makes a great threat against cyberspace security. The attack detection is an important part of the security protection, which provides effective support for defense system through the rapid and accurate identification of attacks. According to the attacker’s different URL of the Web service, the AL-DDoS attack is divided into three categories, including a random URL attack and a fixed and a traverse one. In order to realize identification of attacks, a mapping matrix of the joint entropy vector is constructed. By defining and computing the value of EUPI and jEIPU, a visual coordinate discrimination diagram of entropy vector is proposed, which also realizes data dimension reduction from N to two. In terms of boundary discrimination and the region where the entropy vectors fall in, the class of AL-DDoS attack can be distinguished. Through the study of training data set and classification, the results show that the novel algorithm can effectively distinguish the web server DDoS attack from normal burst traffic.

With the popularity of the Internet and the increase in the level of information technology, cyber attacks have become an increasingly serious problem. They pose a great threat to the security of individuals, enterprises, and the state. This has made network intrusion detection technology critically important. In this paper, a malicious traffic detection model is constructed based on a decision tree classifier of entropy and a proximal policy optimisation algorithm (PPO) of deep reinforcement learning. Firstly, the decision tree idea in machine learning is used to make a preliminary classification judgement on the dataset based on the information entropy. The importance score of each feature in the classification work is calculated and the features with lower contributions are removed. Then, it is handed over to the PPO algorithm model for detection. An entropy regularity term is introduced in the process of the PPO algorithm update. Finally, the deep reinforcement learning algorithm is used to continuously train and update the parameters during the detection process, and finally, the detection model with higher accuracy is obtained. Experiments show that the binary classification accuracy of the malicious traffic detection model based on the deep reinforcement learning PPO algorithm can reach 99.17% under the CIC-IDS2017 dataset used in this paper.