Attention-based networks currently identify their effectiveness in multimodal sentiment analysis. However, existing methods ignore the redundancy of auxiliary modalities. More importantly, existing methods only attend to top-down attention (static process) or down-top attention (implicit process), leading to the coarse-grained multimodal sentiment context. In this paper, during the preprocessing period, we first propose the multimodal dynamic enhanced block to capture the intra-modality sentiment context. This can effectively decrease the intra-modality redundancy of auxiliary modalities. Furthermore, the bi-direction attention block is proposed to capture fine-grained multimodal sentiment context via the novel bi-direction multimodal dynamic routing mechanism. Specifically, the bi-direction attention block first highlights the explicit and low-level multimodal sentiment context. Then, the low-level multimodal context is transmitted to a carefully designed bi-direction multimodal dynamic routing procedure. This allows us to dynamically update and investigate high-level and much more fine-grained multimodal sentiment contexts. The experiments demonstrate that our fusion network can achieve state-of-the-art performance. Notably, our model outperforms the best baseline on the metric ‘Acc-7’ with an improvement of 6.9%.

Multimodal Sentiment Analysis is a promising research area for modeling multiple heterogeneous modalities. Two major challenges that exist in this area are a) multimodal data is unaligned in nature due to the different sampling rates of each modality, and b) long-range dependencies between elements across modalities. These challenges increase the difficulty of conducting efficient multimodal fusion. In this work, we propose a novel end-to-end network named Cross Hyper-modality Fusion Network (CHFN). The CHFN is an interpretable Transformer-based neural model that provides an efficient framework for fusing unaligned multimodal sequences. The heart of our model is to dynamically adjust word representations in different non-verbal contexts using unaligned multimodal sequences. It is concerned with the influence of non-verbal behavioral information at the scale of the entire utterances and then integrates this influence into verbal expression. We conducted experiments on both publicly available multimodal sentiment analysis datasets CMU-MOSI and CMU-MOSEI. The experiment results demonstrate that our model surpasses state-of-the-art models. In addition, we visualize the learned interactions between language modality and non-verbal behavior information and explore the underlying dynamics of multimodal language data.

Sleep staging is important in sleep research since it is the basis for sleep evaluation and disease diagnosis. Related works have acquired many desirable outcomes. However, most of current studies focus on time-domain or frequency-domain measures as classification features using single or very few channels, which only obtain the local features but ignore the global information exchanging between different brain regions. Meanwhile, brain functional connectivity is considered to be closely related to brain activity and can be used to study the interaction relationship between brain areas. To explore the electroencephalography (EEG)-based brain mechanisms of sleep stages through functional connectivity, especially from different frequency bands, we applied phase-locked value (PLV) to build the functional connectivity network and analyze the brain interaction during sleep stages for different frequency bands. Then, we performed the feature-level, decision-level and hybrid fusion methods to discuss the performance of different frequency bands for sleep stages. The results show that (1) PLV increases in the lower frequency band (delta and alpha bands) and vice versa during different stages of non-rapid eye movement (NREM); (2) alpha band shows a better discriminative ability for sleeping stages; (3) the classification accuracy of feature-level fusion (six frequency bands) reaches 96.91% and 96.14% for intra-subject and inter-subjects respectively, which outperforms decision-level and hybrid fusion methods.

Objective. As one of the most popular brain–computer interface (BCI) paradigms, affective BCI (aBCI) decodes the human emotional states from brain signals and imposes necessary feedback to achieve neural regulation when negative emotional states (i.e. depression, anxiety) are detected, which are considered as the two basic functions of aBCI systems. Electroencephalogram (EEG) is the scalp reflection of neural activities and has been regarded as the gold standard of emotional effects. Recently, rapid progresses have been made for emotion recognition and regulation with the purpose of constructing a high-performance closed-loop EEG-based aBCI system. Therefore, it is necessary to make a timely review for aBCI research by summarizing the current progresses as well as challenges and opportunities, to draw the attention from both academia and industry. Toward this goal, a systematic literature review was performed to summarize not only the recent progresses in emotion recognition and regulation from the perspective of closed-loop aBCI, but also the main challenges and future research focuses to narrow the gap between the current research and real applications of aBCI systems. Approach. A systematic literature review on EEG-based emotion recognition and regulation was performed on Web of Science and related databases, resulting in more than 100 identified studies. These studies were analyzed according to the experimental paradigm, emotion recognition methods in terms of different scenarios, and the applications of emotion recognition in diagnosis and regulation of affective disorders. Main results. Based on the literature review, advancements for EEG-based aBCI research were extensively summarized from six aspects including the ‘emotion elicitation paradigms and data sets’, ‘inner exploration of EEG information’, ‘outer extension of fusing EEG with other data modalities’, ‘cross-scene emotion recognition’, ‘emotion recognition by considering real scenarios’, and ‘diagnosis and regulation of affective disorders’. In addition, future opportunities were concluded by focusing on the main challenges in hindering the aBCI system to move from laboratory to real applications. Moreover, the neural mechanisms and theoretical basis behind EEG emotion recognition and regulation are also introduced to provide support for the advancements and challenges in aBCI. Significance. This review summarizes the current practices and performance outcomes in emotion recognition and regulation. Future directions in response to the existing challenges are provided with the expectation of guiding the aBCI research to focus on the necessary key technologies of aBCI systems in practical deployment.