For a large number of network attacks, feature selection is used to improve intrusion detection efficiency. A new mutual information algorithm of the redundant penalty between features (RPFMI) algorithm with the ability to select optimal features is proposed in this paper. Three factors are considered in this new algorithm: the redundancy between features, the impact between selected features and classes and the relationship between candidate features and classes. An experiment is conducted using the proposed algorithm for intrusion detection on the KDD Cup 99 intrusion dataset and the Kyoto 2006+ dataset. Compared with other algorithms, the proposed algorithm has a much higher accuracy rate (i.e., 99.772%) on the DOS data and can achieve better performance on remote-to-login (R2L) data and user-to-root (U2R) data. For the Kyoto 2006+ dataset, the proposed algorithm possesses the highest accuracy rate (i.e., 97.749%) among the other algorithms. The experiment results demonstrate that the proposed algorithm is a highly effective feature selection method in the intrusion detection.

Since the rapid development of the internet, the emergence of network intrusion has become the focus of studies for scholars and security enterprises. As an important device for detecting and analyzing malicious behaviors in networks, IDS (Intrusion Detection Systems) is widely deployed in enterprises, organizations and plays a very important role in cyberspace security. The massive log data produced by IDS not only contains information about intrusion behaviors but also contains potential intrusion patterns. Through normalizing, correlating, and modeling data, we can obtain the patterns of different intrusion scenarios. Based on the previous works in the area of alert correlation and analyzing, this paper proposed a framework named IACF (Intrusion Action Based Correlation Framework), which improved the process of alert aggregating, action extraction, and scenario discovery, and applied a novel method for extracting intrusion sessions based on temporal metrics. The proposed framework utilized a new grouping method for raw alerts based on the concept of intrinsic strong correlations, rather than the conventional time windows and hyper alerts. For discovering high stable correlations between actions, redundant actions and action link modes are removed from sessions by a pruning algorithm to reduce the impact of false positives, finally, a correlation graph is constructed by fusing the pruned sessions, based on the correlation graph, a prediction method for the future attack is proposed. The experiment result shows that the framework is efficient in alert correlation and intrusion scenario construction.

With the continuous development of network technology, an intrusion detection system needs to face detection efficiency and storage requirement when dealing with large data. A reasonable way of alleviating this problem is instance selection, which can reduce the storage space and improve intrusion detection efficiency by selecting representative instances. An instance is representative not only in its class but also in different classes. This representativeness reflects the importance of an instance. Since the existing instance selection algorithm does not take into account the above situations, some selected instances are redundant and some important instances are removed, increasing storage space and reducing efficiency. Therefore, a new representativeness of instance is proposed and considers not only the influence of all instances of the same class on the selected instance but also the influence of instances of different classes on the selected instance. Moreover, it considers the influence of instances of different classes as an advantageous factor. Based on this representativeness, two instance selection algorithms are proposed to handle balanced and imbalanced data problems for intrusion detection. One is a representative-based instance selection for balanced data, which is named RBIS and selects the same proportion of instances from each class. The other is a representative-based instance selection for imbalanced data, which is named RBIS-IM and selects important majority instances according to the number of instances of the minority class. Compared with other algorithms on the benchmark data sets of intrusion detection, experimental results verify the effectiveness of the proposed RBIS and RBIS-IM algorithms and demonstrate that the proposed algorithms can achieve a better balance between accuracy and reduction rate or between balanced accuracy and reduction rate.

With the development of intrusion detection, a number of the intelligence algorithms (e.g., artificial neural networks) are introduced to enhance the performance of the intrusion detection systems. However, many intelligence algorithms should be trained before being used, and retrained regularly, which is not applicable for continuous online learning and analyzing. In this paper, a new online intrusion scenario discovery framework is proposed and the intelligence algorithm HTM (Hierarchical Temporal Memory) is employed to improve the performance of the online learning ability of the system. The proposed framework can discover and model intrusion scenarios, and the constructed model keeps evolving with the variance of the data. Additionally, a series of data preprocessing methods are introduced to enhance its adaptability to the noisy and twisted data. The experimental results show that the framework is effective in intrusion scenario discovery, and the discovered scenario is more concise and accurate than our previous work.

In the algorithm model training, the scale of the labeled data and the type of algorithm model determines the accuracy of algorithm model. In reality, when training a new system, the amount of unlabeled data is often much larger than that of labeled data. There are linear models, matrix decomposition models and so on for the selection of recommended algorithm models. Given the above problems, consider using the semi-supervised co-training method to construct a dual-view hybrid model algorithm framework through co-training of recommendation algorithm models based on different mathematical foundations. The algorithm framework is verified on the movie data set MovieLens. The experimental results show that the semi-supervised co-training recommendation algorithm based on the hybrid model reduces the root-mean-square error by 1.4%, the accuracy of recommendation results is improved.