Insider threats pose significant risks to organizational security, often going undetected due to their familiarity with the systems. Detection of insider threats faces challenges of imbalanced data distributions and difficulties in fine-grained detection. Specifically, anomalous users and anomalous behaviors take up a very small fraction of all insider behavior data, making precise detection of anomalous users challenging. Moreover, not all behaviors of anomalous users are anomalous, so it is difficult to detect their behaviors by standardizing with single rules or models. To address these challenges, this paper presents a novel approach for insider threat detection, leveraging machine learning techniques to conduct multi-granularity anomaly detection. We introduce the Multi-Granularity User Anomalous Behavior Detection (MG-UABD) system, which combines coarse-grained and fine-grained anomaly detection to improve the accuracy and effectiveness of detecting anomalous behaviors. The coarse-grained module screens all of the user activities to identify potential anomalies, while the fine-grained module focuses on specific anomalous users to refine the detection process. Besides, MG-UABD employs a combination of oversampling and undersampling techniques to address the imbalance in the datasets, ensuring robust model performance. Through extensive experimentation on the commonly used dataset CERT R4.2, we demonstrate that the MG-UABD system achieves superior detection rate and precision. Compared to the suboptimal model, the accuracy has increased by 3.1% and the detection rate has increased by 4.1%. Our findings suggest that a multi-granularity approach for anomaly detection, combined with tailored sampling strategies, is highly effective in addressing insider threats.

In cyberspace, intrusion and detection constitute dynamic and continuous game processes, where data streams are generated incrementally during intrusion. To mitigate intrusions and safeguard assets effectively, it is imperative to take prompt actions based on real-time detection and analysis of the currently available data streams. However, existing approaches that rely on complete and clean data struggle to keep pace with the continuous real-time flow of new network data. To address this issue, we introduce IL-IDS (Incremental Learning for Intrusion Detection Systems), a novel intrusion detection approach that utilizes incremental learning to enable accurate and timely detection of intrusions in real-world scenarios, where the need for real-time processing and learning from newly generated traffic data is paramount. IL-IDS performs in scenarios with limited data availability, where it initially transforms textual data streams into vectorized representations and leverages a variation autoencoder (VAE) to compress these vectors, efficiently extracting their latent features. Then a classifier is trained to distinguish attack and normal behaviors, and a three-way decision method is employed to establish a boundary for ambiguous data that pose challenges in direct classification. Concurrently, threat intelligence is integrated into this process to enhance the accuracy of decision-making. We validate the effectiveness and efficiency of IL-IDS with experiments on real-world deployments during an international activity, highlighting its robustness and reliability in intrusion detection applications, especially under conditions of confined data streams. Notably, IL-IDS has exhibited comparable accuracy and recall results, and attains exceptional 99.93% precision and 96.83% F1-score, which demonstrates a notable improvement of 5.27% and 2.59% respectively in comparison to intrusion detection models trained on complete and readily available data.

Insider threats pose significant risks to organizational security, often going undetected due to their familiarity with the systems. Detection of insider threats faces challenges of imbalanced data distributions and difficulties in fine-grained detection. Specifically, anomalous users and anomalous behaviors take up a very small fraction of all insider behavior data, making precise detection of anomalous users challenging. Moreover, not all behaviors of anomalous users are anomalous, so it is difficult to detect their behaviors standardizing with single rules or models. To address these challenges, This paper presents a novel approach for insider threat detection, leveraging machine learning techniques to analyze diverse behavior patterns within an organization. We introduce the Multi-Granularity User Anomalous Behavior Detection (MG-UABD) system, which combines coarse-grained and fine-grained anomaly detection to improve the accuracy and effectiveness of detecting anomalous behaviors. The coarse-grained module screens all of the user activities to identify potential anomalies, while the fine-grained module focuses on specific anomalous users to refine the detection process. Besides, MG-UABD employs a combination of over-sampling and under-sampling techniques to address the imbalance in the datasets, ensuring robust model performance. Through extensive experimentation on the commonly used dataset CERT R4.2, we demonstrate that the MG-UABD system achieves superior detection rate and precision, particularly in identifying anomalies in Internet usage and file operations. Our findings suggest that a multi-granularity approach for anomaly detection, combined with tailored sampling strategies, is highly effective in addressing insider threats.