Applications of machine learning are subject to three major components that contribute to the final performance metrics. Within the category of neural networks, and deep learning specifically, the first two are the architecture for the model being trained and the training approach used. This work focuses on the third component, the data used during training. The primary questions that arise are “what is in the data” and “what within the data matters?” looking into the radio frequency machine learning (RFML) field of automatic modulation classification (AMC) as an example of a tool used for situational awareness, the use of synthetic, captured, and augmented data are examined and compared to provide insights about the quantity and quality of the available data necessary to achieve desired performance levels. Three questions are discussed within this work: (1) how useful a synthetically trained system is expected to be when deployed without considering the environment within the synthesis, (2) how can augmentation be leveraged within the RFML domain, and, lastly, (3) what impact knowledge of degradations to the signal caused by the transmission channel contributes to the performance of a system. In general, the examined data types each make useful contributions to a final application, but captured data germane to the intended use case will always provide more significant information and enable the greatest performance. Despite the benefit of captured data, the difficulties and costs that arise from live collection often make the quantity of data needed to achieve peak performance impractical. This paper helps quantify the balance between real and synthetic data, offering concrete examples where training data is parametrically varied in size and source.
Generating high-quality, real-world, well-labeled datasets for radio frequency machine learning (RFML) applications often proves prohibitively cumbersome and expensive, leading to the lowavailability of high-fidelity, low-cost datasets. Specific emitter identification (SEI) in particular requires a hardware setup capable of supporting transmitting using many different radios, while automated modulation classification (AMC) performance is primarily driven by SNR, channel effects, and the similarity of modulation types. These factors give rise to the need for scalable methods of inexpensive dataset generation. This paper describes the design considerations and a proof-of-concept implementation of a blind user reconfigurable platform capable of creating SEI and AMC datasets throughout a variety of real-world conditions. This paper additionally describes the reliability and performance of the platform relative to existing real-world data generation methods and compares generated datasets to those already present in the literature. This work also describes the software post-processing steps taken to isolate, label, and cull captured data and transform these into a high-quality dataset.
The data used during training in any given application space are directly tied to the performance of the system once deployed. While there are many other factors that are attributed to producing high-performance models based on the Neural Scaling Law within Machine Learning, there is no doubt that the data used to train a system provide the foundation from which to build. One of the underlying heuristics used within the Machine Learning space is that having more data leads to better models, but there is no easy answer to the question, “How much data is needed to achieve the desired level of performance?” This work examines a modulation classification problem in the Radio Frequency domain space, attempting to answer the question of how many training data are required to achieve a desired level of performance, but the procedure readily applies to classification problems across modalities. The ultimate goal is to determine an approach that requires the lowest amount of data collection to better inform a more thorough collection effort to achieve the desired performance metric. By focusing on forecasting the performance of the model rather than the loss value, this approach allows for a greater intuitive understanding of data volume requirements. While this approach will require an initial dataset, the goal is to allow for the initial data collection to be orders of magnitude smaller than what is required for delivering a system that achieves the desired performance. An additional benefit of the techniques presented here is that the quality of different datasets can be numerically evaluated and tied together with the quantity of data, and ultimately, the performance of the architecture in the problem domain.