Intrusion detection systems (IDSs) play a significant role in the field of network security, dealing with the ever-increasing number of network threats. Machine learning-based IDSs have attracted a lot of interest owing to their powerful data-driven learning capabilities. However, it is challenging to train the supervised learning algorithms when there are no attack data at hand. Semi-supervised anomaly detection algorithms, which train the model with only normal data, are more suitable. In this study, we propose a novel semi-supervised anomaly detection-based IDS that leverages the capabilities of representation learning and two anomaly detectors. In detail, the autoencoder (AE) is applied to extract representative features of normal data in the first step, and then two semi-supervised detectors, the one-class support vector machine (OCSVM) and Gaussian mixture model (GMM), are trained on the derived features. The two detectors collaborate to detect anomalous samples. The OCSVM predicts the abnormal samples initially, and after that, the GMM is applied to recheck the misclassified samples further. The experiments demonstrate that the AE improves the detection rate, and two detectors are more promising than a single one.

Tor is the most widely used anonymous communication network. Tor has developed a series of pluggable transports (PTs) to obfuscate traffic and avoid censorship. These PTs use different traffic obfuscation techniques, and many of them have been maintained and updated. In order to achieve continual learning against PTs and their updates, this paper proposes an incremental learning model for Tor traffic detection. First, we analyzed several common traffic obfuscation techniques, including randomization, mimicry, and tunneling. A feature set was designed for Tor obfuscation traffic detection. Second, this paper constructs the Tor incremental learning framework and proposes edge exemplar enhancement to enhance the memory of trained models for previous classes. It can enhance the previous class memory of the model through edge feature enhancement and selective replay to alleviate the catastrophic forgetting problem of incremental learning. Finally, we combined public and self-collected datasets to simulate the development of Tor PTs and verify the effectiveness of our model. The experimental results show that the improved model in this paper has the highest accuracy rate of 87.6% in the simulated environment. This means that the incremental learning model can effectively cope with the updating of PTs.