Sleep scoring involves the inspection of multimodal recordings of sleep data to detect potential sleep disorders. Given that symptoms of sleep disorders may be correlated with specific sleep stages, the diagnosis is typically supported by the simultaneous identification of a sleep stage and a sleep disorder. This paper investigates the automatic recognition of sleep stages and disorders from multimodal sensory data (EEG, ECG, and EMG). We propose a new distributed multimodal and multilabel decision-making system (MML-DMS). It comprises several interconnected classifier modules, including deep convolutional neural networks (CNNs) and shallow perceptron neural networks (NNs). Each module works with a different data modality and data label. The flow of information between the MML-DMS modules provides the final identification of the sleep stage and sleep disorder. We show that the fused multilabel and multimodal method improves the diagnostic performance compared to single-label and single-modality approaches. We tested the proposed MML-DMS on the PhysioNet CAP Sleep Database, with VGG16 CNN structures, achieving an average classification accuracy of 94.34% and score of 0.92 for sleep stage detection (six stages) and an average classification accuracy of 99.09% and score of 0.99 for sleep disorder detection (eight disorders). A comparison with related studies indicates that the proposed approach significantly improves upon the existing state-of-the-art approaches.
Existing automatic sleep stage detection methods predominantly use convolutional neural network classifiers (CNNs) trained on features extracted from single-modality signals such as electroencephalograms (EEG). On the other hand, multimodal approaches propose very complexly stacked network structures with multiple CNN branches merged by a fully connected layer. It leads to very high computational and data requirements. This study proposes replacing a stacked network with a distributed neural network system for multimodal sleep stage detection. It has relatively low computational and training data requirements while providing highly competitive results. The proposed multimodal classification and decision-making system (MM-DMS) method applies a fully connected shallow neural network, arbitrating between classification outcomes given by an assembly of independent convolutional neural networks (CNNs), each using a different single-modality signal. Experiments conducted on the CAP Sleep Database data, including the EEG-, ECG-, and EMG modalities representing six stages of sleep, show that the MM-DMS significantly outperforms each single-modality CNN. The fully-connected shallow network arbitration included in the MM-DMS outperforms the traditional majority voting-, average probability-, and maximum probability decision-making methods.