31 October 2023 in IEEE Transactions on Biomedical Engineering
Nowadays, how to estimate vigilance with higher accuracy has become a hot field of research direction. Although the increasing available modalities opens the door for amazing new possibilities to achieve good performance, the uncertain cross-modal interaction still poses a real challenge to the multimodal fusion. In this paper, a cross-modality alignment method has been proposed based on the contrastive learning for extracting shared but not the same information among modalities. The contrastive learning is adopted to minimize the intermodal differences by maximizing the similarity of semantic representation of modalities. Applying our proposed modeling framework, we evaluated our approach on SEED-VIG dataset consisting of EEG and EOG signals. Experiments showed that our study achieved state-of-the-art multimodal vigilance estimation performance both in intra-subject and inter-subject situations, the average of RMSE/CORR were improved to 0.092/0.893 and 0.144/0.887, respectively. In addition, analysis on the frequency bands showed that theta and alpha activities contain valuable information for vigilance estimation, and the correlation between them and PERCLOS can be significantly improved by contrastive learning. We argue that the proposed method in the inter-subject case could offer the possibility of reducing the high-cost of data annotation, and further analysis may provide an idea for the application of multimodal vigilance regression.
6 October 2025 in IEEE Transactions on Affective Computing
Micro-expression recognition (MER), a critical subfield of affective computing, presents greater challenges than macro-expression recognition due to its brief duration and low intensity. While incorporating prior knowledge has been shown to enhance MER performance, existing methods predominantly rely on simplistic, singular sources of prior knowledge, failing to fully exploit multi-source information. This paper introduces the Multi-Prior Fusion Network (MPFNet), leveraging a progressive training strategy to optimize MER tasks. We propose two complementary encoders: the Generic Feature Encoder (GFE) and the Advanced Feature Encoder (AFE), both based on Inflated 3D ConvNets (I3D) with Coordinate Attention (CA) mechanisms, to improve the model's ability to capture spatiotemporal and channel-specific features. Inspired by developmental psychology, we present two variants of MPFNet—MPFNet-P and MPFNet-C—corresponding to two fundamental modes of infant cognitive development: parallel and hierarchical processing. These variants enable the evaluation of different strategies for integrating prior knowledge. Extensive experiments demonstrate that MPFNet significantly improves MER accuracy while maintaining balanced performance across categories, achieving accuracies of 0.811, 0.924, and 0.857 on the SMIC, CASME II, and SAMM datasets, respectively. To the best of our knowledge, our approach achieves state-of-the-art performance on the SMIC and SAMM datasets. The source code is available at: https://github.com/Mac0504/MPFNet.
9 December 2025 in Information Fusion
1 May 2026 in IEEE Transactions on Biomedical Engineering
Electroencephalography (EEG) has emerged as a powerful tool for modeling human brain states. However, the widespread adoption of EEG-based recognition systems is hindered by low signal-to-noise ratios and the scarcity of labeled data. While existing studies often tackle these challenges in isolation, we propose a novel Cross-Device Representation Consistency (CDRC) pretraining paradigm that addresses both issues simultaneously. CDRC leverages self-supervised signals derived from representation distances and is trained through contrastive estimation. Specifically, our approach employs a transformer based dual-branch single-view embedding prediction task, combining with a contrastive feature alignment module to extract robust and discriminative representations. We first evaluate the CDRC model on a low signal-to-noise ratio emotion classification task involving wearable dry electrodes. Furthermore, we extend CDRC to a multimodal fusion setting to address a cross-device vigilance regression task involving heterogeneous physiological modalities. Extensive experiments on the PaDWEED and SEED-VIG datasets demonstrate that CDRC achieves performance comparable to fully supervised methods and reaches the stat-of-the-art results of existing self-supervised methods, setting a new benchmark in this field. Notably, its strong performance on subject-independent tasks highlights its effectiveness in mitigating subject variability. These results underscore the potential of CDRC to significantly enhance the practicality and scalability of EEG-based recognition systems, marking a meaningful step toward real-world brain-computer interfaces.