Existing machine learning-based malware traffic recognition techniques can effectively detect abnormal behaviors in the network. However, almost all of them focus on a closed-set scenario in which the data used for training and testing come from the same label space. Since sophisticated malware and advanced persistent threats are evolving, it is impossible to exhaust all attacks to train a complete recognition model under the existing technical conditions. Therefore, recognition in the real network is an open-set problem, i.e., the recognition system should identify unknown and unseen attacks at test time. In this paper, we propose an uncertainty-aware method to identify known malicious traffic accurately and handle unknown traffic effectively. This method employs predictive uncertainty in deep learning as an indicator for unknown class detection. The predictive uncertainty represents the confidence in neural network predictions. In particular, the Deep Evidence Malware Traffic Recognition (DEMTR) model is presented to provide the multi-classification probability and predictive uncertainty in open-set scenarios using evidential deep learning. We demonstrate the performance of DEMTR on the MCFP dataset. Experimental results indicate that the proposed model outperforms the baseline methods in accuracy and F₁-score.

Deep learning has achieved good classification results in the field of traffic classification in recent years due to its good feature representation ability. However, the existing traffic classification technology cannot meet the requirements for the incremental learning of tasks in online scenarios. In addition, due to the high concealment and fast update speed of malicious traffic, the number of labeled samples that can be captured is scarce, and small samples cannot drive neural network training, resulting in poor performance of the classification model. Therefore, this paper proposes an incremental learning method for small-sample malicious traffic classification. The method uses the pruning strategy to find the redundant network structure and dynamically allocates redundant neurons for training based on the proposed measurement method according to the difficulty of the new class. This enables the network to perform incremental learning without excessively consuming storage and computing resources, and reasonable allocation improves the classification accuracy of new classes. At the same time, through the knowledge transfer method, the model can reduce the catastrophic forgetting of the old class, relieve the pressure of training large parameters with small-sample data, and improve the model classification performance. Experiments involving multiple datasets and settings show that our method is superior to the established baseline in terms of classification accuracy, consuming 50% less memory.

Deep learning is successful in providing adequate classification results in the field of traffic classification due to its ability to characterize features. However, malicious traffic captures insufficient data and identity tags, which makes it difficult to reach the data volume required to drive deep learning. The problem of classifying small-sample malicious traffic has gradually become a research hotspot. This paper proposes a small-sample malicious traffic classification method based on deep transfer learning. The proposed DA-Transfer method significantly improves the accuracy and efficiency of the small-sample malicious traffic classification model by integrating both data and model transfer adaptive modules. The data adaptation module promotes the consistency of the distribution between the source and target datasets, which improves the classification performance by adaptive training of the prior model. In addition, the model transfer adaptive module recommends the transfer network structure parameters, which effectively improves the network training efficiency. Experiments show that the average classification accuracy of the DA-Transfer method reaches 93.01% on a small-sample dataset with less than 200 packets per class. The training efficiency of the DA-Transfer model is improved by 20.02% compared to traditional transfer methods.