Host-based threats such as Program Attack, Malware Implantation, and Advanced Persistent Threats (APT), are commonly adopted by modern attackers. Recent studies propose leveraging the rich contextual information in data provenance to detect threats in a host. Data provenance is a directed acyclic graph constructed from system audit data. Nodes in a provenance graph represent system entities (e.g., processes and files ) and edges represent system calls in the direction of information flow. However, previous studies, which extract features of the provenance graph, are not sensitive to the small quantity of threat-related entities and thus result in low performance when hunting stealthy threats. We present THREATRACE, an anomaly-based detector that detects host-based threats at system entity level without prior knowledge of attack patterns. We tailor GraphSAGE, an inductive graph neural network, to learn every benign entity’s role in a provenance graph. THREATRACE is a real-time system, which is scalable of monitoring a long-term running host and capable of detecting host-based intrusion in their early phase. We evaluate THREATRACE on five public datasets. The results show that THREATRACE outperforms seven state-of-the-art host intrusion detection systems.
Deep learning (DL) based anomaly detection has shown great promise in the field of security due to its remarkable performance in various tasks. However, the issue of poor interpretability in DL models has significantly impeded their deployment in practical security applications. Despite the progress made in existing studies on DL explanations, the majority of them focus on providing local explanations for individual samples, neglecting the global understanding of the model knowledge. Furthermore, most explanations for supervised models fail to apply to anomaly detection due to their different learning mechanisms. In this work, we address the gap in the existing research by proposing GEAD, a novel global explanation for DL-based anomaly detection, to extract high-fidelity rules from DL models. We apply GEAD to two security applications, network intrusion detection and system log anomaly detection, and demonstrate the efficacy with three usages: comparing model knowledge with expert knowledge, identifying knowledge discrepancies between models, and combining model and expert knowledge. We provide several case studies to showcase how GEAD can significantly enhance existing anomaly detection systems. Moreover, we provide a real-world deployment in a SCADA system to showcase the potential in practice. Some important insights are drawn to help the community understand and improve anomaly detection systems in security.
Recently enterprises and governments face escalating APT attacks, leading to significant economic losses. APT attacks often persist for extended periods, necessitating the storage of extensive audit logs for effective detection. To reduce data storage overhead, enterprises commonly adopt compression strategies. However, efficient compression strategies may introduce additional query overhead. Existing approaches propose data reduction algorithms, but these methods can compromise data integrity, rendering current attack investigation and anomaly-based intrusion detection ineffective. To address these difficulties, we present AudiTrim, a system that ensures real-time, general, efficient, and low-overhead data compaction without compromising attack investigation and anomaly-based intrusion detection. It efficiently reduces log sizes without impacting user experiences, achieving real-time compaction and adaptable deployment on different operating systems. AudiTrim employs two strategies: 1) Data Reduction: By analyzing the types of duplicate edges, our data reduction approach not only considers a broader range of scenarios for redundant edges compared to previous methods but also enhances the efficiency of data reduction. 2) Data Compression: By aggregating log information at the server-side and training a compression model, we facilitate a data compression algorithm that ensures ease of querying. Both strategies meet real-time, low-overhead, and general requirements, fulfilling enterprise data storage needs. The final compaction ratio reaches 26×-65×.
The provenance graph, due to its ability to fully preserve historical information within a host, has been extensively studied and applied in the field of host intrusion detection. Such research is collectively referred to as provenance-based intrusion detection systems (PIDSes). However, recent surveys on PIDSes face limitations such as outdated literature and insufficiently comprehensive classifications. To systematically summarize the evolutionary trajectory and experimental evaluation framework of PIDSes, this paper conducts a literature review from the perspective of PIDSes evolution, proposing a more refined framework for host threat detection. For each fundamental step within the framework, representative PIDSes are reviewed and introduced. Additionally, we qualitatively analyze state-of-the-art (SOTA) PIDSes from the perspective of experimental evaluation, summarize seven key findings derived from experimental evaluations, and provide an analysis of future research directions addressing some existing limitations in the field of PIDSes.
Recent studies in Vision-and-Language Navigation (VLN) train RL agents to execute natural-language navigation instructions in photorealistic environments, as a step towards robots that can follow human instructions. However, given the scarcity of human instruction data and limited diversity in the training environments, these agents still struggle with complex language grounding and spatial language understanding. Pretraining on large text and image-text datasets from the web has been extensively explored but the improvements are limited. We investigate large-scale augmentation with synthetic instructions. We take 500+ indoor environments captured in densely-sampled 360 degree panoramas, construct navigation trajectories through these panoramas, and generate a visually-grounded instruction for each trajectory using Marky, a high-quality multilingual navigation instruction generator. We also synthesize image observations from novel viewpoints using an image-to-image GAN. The resulting dataset of 4.2M instruction-trajectory pairs is two orders of magnitude larger than existing human-annotated datasets, and contains a wider variety of environments and viewpoints. To efficiently leverage data at this scale, we train a simple transformer agent with imitation learning. On the challenging RxR dataset, our approach outperforms all existing RL agents, improving the state-of-the-art NDTW from 71.1 to 79.1 in seen environments, and from 64.6 to 66.8 in unseen test environments. Our work points to a new path to improving instruction-following agents, emphasizing large-scale imitation learning and the development of synthetic instruction generation capabilities.