Most emotion recognition systems still present limited applicability to new users due to the inter-subject variability of electroencephalography (EEG) signals. Although domain adaptation methods have been adopted to tackle this problem, most methodologies deal with unlabeled data from a target subject. However, a few labeled samples from a target subject could also be included to boost cross-subject emotion recognition. In this paper, we present a semi-supervised domain adaptation framework to align the joint distributions of subjects, assuming that fine-grained structures must be aligned to perform a greater knowledge transfer. To achieve this, the proposed framework performs a multi-source alignment of features at subject level, while predictions are aligned over the global feature space. To support joint distribution alignment, inter-class separation and consistent predictions are ensured on the target subject. We perform experiments using two public benchmark datasets, SEED and SEED-IV, with two different sampling strategies to incorporate a few labeled samples from a target subject. Our proposal achieves an average accuracy of 93.55% and 87.96% on SEED and SEED-IV, using three labeled target samples of each class. Moreover, we obtained an average accuracy of 91.79% and 85.45% on SEED and SEED-IV by incorporating 10 labeled samples from the first EEG trial of each class.

Multi-modal classifiers for emotion recognition have become prominent, as the emotional states of subjects can be more comprehensively inferred from Electroencephalogram (EEG) signals and eye movements. However, existing classifiers experience a decrease in performance due to the distribution shift when applied to new users. Unsupervised domain adaptation (UDA) emerges as a solution to address the distribution shift between subjects by learning a shared latent feature space. Nevertheless, most UDA approaches focus on a single modality, while existing multi-modal approaches do not consider that fine-grained structures should also be explicitly aligned and the learned feature space must be discriminative. In this paper, we propose Coarse and Fine-grained Distribution Alignment with Correlated and Separable Features (CFDA-CSF), which performs a coarse alignment over the global feature space, and a fine-grained alignment between modalities from each domain distribution. At the same time, the model learns intra-domain correlated features, while a separable feature space is encouraged on new subjects. We conduct an extensive experimental study across the available sessions on three public datasets for multi-modal emotion recognition: SEED, SEED-IV, and SEED-V. Our proposal effectively improves the recognition performance in every session, achieving an average accuracy of 93.05%, 85.87% and 91.20% for SEED; 85.72%, 89.60%, and 86.88% for SEED-IV; and 88.49%, 91.37% and 91.57% for SEED-V.

Multimodal emotion recognition from electroencephalogram (EEG) and eye movement signals has shown to be a promising approach to provide more discriminative information about human emotional states. However, most current works rely on a subject-dependent approach, limiting their applicability to new users. Recently, some studies have explored multimodal domain adaptation to address the mentioned issue by transferring information from known subjects to new ones. Unfortunately, existing methods are still exposed to negative transfer as a suboptimal distribution alignment is performed between subjects, while irrelevant information is not discarded. In this article, we present a multimodal and multisource domain adaptation (MMDA) method, which adopts the following three strategies: 1) marginal and conditional distribution alignments must be performed between each known subject and a new one; 2) relevant distribution alignments must be prioritized to avoid a negative transfer; and 3) modality fusion results should be improved by extracting more discriminative features from EEG signals and selecting relevant features across modalities. Our proposed method was evaluated with leave-one-subject-out cross validation on four public datasets: SEED, SEED-GER, SEED-IV, and SEED-V. Experimental results show that our proposal outperforms state-of-the-art results for each dataset when subject data from different sessions are combined into a single dataset. Moreover, MMDA exceeds the state of the art in 8 out of 11 different sessions when each session is evaluated.