This paper presents a novel deep-learning (DL)-based approach for classifying digitally modulated signals, which involves the use of capsule networks (CAPs) together with the cyclic cumulant (CC) features of the signals. These were blindly estimated using cyclostationary signal processing (CSP) and were then input into the CAP for training and classification. The classification performance and the generalization abilities of the proposed approach were tested using two distinct datasets that contained the same types of digitally modulated signals, but had distinct generation parameters. The results showed that the classification of digitally modulated signals using CAPs and CCs proposed in the paper outperformed alternative approaches for classifying digitally modulated signals that included conventional classifiers that employed CSP-based techniques, as well as alternative DL-based classifiers that used convolutional neural networks (CNNs) or residual networks (RESNETs) with the in-phase/quadrature (I/Q) data used for training and classification.

Machine learning has become a powerful tool for solving problems in various engineering and science areas, including the area of communication systems. This paper presents the use of capsule networks for classification of digitally modulated signals using the I/Q signal components. The generalization ability of a trained capsule network to correctly classify the classes of digitally modulated signals that it has been trained to recognize is also studied by using two different datasets that contain similar classes of digitally modulated signals but that have been generated independently. Results indicate that the capsule networks are able to achieve high classification accuracy. However, these networks are susceptible to the datashift problem which will be discussed in this paper.