We introduce HackSynth, a novel Large Language Model (LLM)-based agent capable of autonomous penetration testing. HackSynth's dual-module architecture includes a Planner and a Summarizer, which enable it to generate commands and process feedback iteratively. To benchmark HackSynth, we propose two new Capture The Flag (CTF)-based benchmark sets utilizing the popular platforms PicoCTF and OverTheWire. These benchmarks include two hundred challenges across diverse domains and difficulties, providing a standardized framework for evaluating LLM-based penetration testing agents. Based on these benchmarks, extensive experiments are presented, analyzing the core parameters of HackSynth, including creativity (temperature and top-p) and token utilization. Multiple open source and proprietary LLMs were used to measure the agent's capabilities. The experiments show that the agent performed best with the GPT-4o model, better than what the GPT-4o's system card suggests. We also discuss the safety and predictability of HackSynth's actions. Our findings indicate the potential of LLM-based agents in advancing autonomous penetration testing and the importance of robust safeguards. HackSynth and the benchmarks are publicly available to foster research on autonomous cybersecurity solutions.

Large Language Models (LLMs), such as GPT-4 and Llama 2, show remarkable proficiency in a wide range of natural language processing (NLP) tasks. Despite their effectiveness, the high costs associated with their use pose a challenge. We present LlamBERT, a hybrid approach that leverages LLMs to annotate a small subset of large, unlabeled databases and uses the results for fine-tuning transformer encoders like BERT and RoBERTa. This strategy is evaluated on two diverse datasets: the IMDb review dataset and the UMLS Meta-Thesaurus. Our results indicate that the LlamBERT approach slightly compromises on accuracy while offering much greater cost-effectiveness.

Radio spectrum is a limited national resource whose efficient utilization is of strategic importance. With the rapid advancement of wireless technologies, maintaining spectrum cleanliness and enabling fast and reliable anomaly detection have become critical challenges. Artificial intelligence (AI)-based approaches have recently shown great promise in addressing these issues; however, their effectiveness strongly depends on the availability of high-quality, representative, and annotated datasets. Generating such datasets is a complex task, further complicated by environmental conditions such as weather. Hungary’s nationwide spectrum monitoring network enables continuous observation of frequency bands, thereby providing the opportunity to construct large-scale and sustainable datasets. This study introduces a measurement methodology designed for the FM sound broadcasting in the VHF band (87.5–108 MHz), presents the resulting dataset, and details the annotation process. The published, openly accessible dataset is expected to serve not only as a valuable reference point but also as a benchmark for the international research community, facilitating the development, validation, and objective comparison of AI-driven spectrum monitoring solutions.

This paper presents a study of three machine-learning approaches for anomaly analysis in spectrum-monitoring data from VHF Band II radio broadcasting. Spectrum anomalies are defined as deviations from expected spectral behavior, including unexpected signal emissions, interference, and irregular variations in signal characteristics. A supervised YOLO-based object detector was evaluated for anomaly localization using independently produced expert annotations, while convolutional autoencoder-based clustering and long short-term memory (LSTM) prediction were investigated for exploratory spectral and temporal analysis, respectively. The supervised experiments examined both within-annotator and cross-annotator performance, whereas the unsupervised and temporal approaches were assessed qualitatively. The autoencoder-based approach revealed recurring spectral structures in latent space, and the LSTM model identified deviations from the recent behavior of individual broadcast transmissions. The principal quantitative results showed within-annotator $F_1$ scores ranging from 0.85 to 0.94, compared with 0.34 to 0.77 in cross-annotator evaluations. These findings indicate that the reported anomaly localization performance is sensitive to the reference annotations, highlighting the importance of annotation consistency and independent evaluation in spectrum-monitoring applications.