Multi-modal continuous affect recognition is becoming an increasingly attractive research topic in affective computing. Previous works mainly focus on modeling the temporal dependency within a sensor modality. To improve system performance most of them adopt early or late fusion in multi-modal continuous affect recognition. However, early fusion suffers from the curse of dimensionality, and late fusion ignores the complementarity and redundancy between multiple streams from different modalities. In this paper, we first introduce the transformer encoder with a self-attention mechanism to temporal dependencies affective state time series and propose a Convolutional Neural Network-Transformer Encoder (CNN-TE) framework for single modality affect recognition. To effectively consider the complementarity and redundancy between multiple streams from different modalities we propose a Transformer Encoder with Multi-modal Multi-head Attention (TEMMA) module designed to progressively refine the inter-modality interactions and intra-modality temporal dependencies simultaneously. We propose a CNN-TEMMA framework for multi-modal affect recognition, which consists of (i)~a 1D CNN based input embedding sub-network to extract local temporal features via causal convolution; (ii)~a TEMMA sub-network to model the multi-modal interactions and temporal dynamics with spatial and temporal attention, to learn high-level representations for each modality; (iii)~an inference sub-network with fully connected layers to estimate the affective state, using as inputs the learned multi-modal representations. The whole framework is trained in an end-to-end fashion. We demonstrate the effectiveness of our proposed framework on the AVEC2016 dataset. Our experimental results show that the proposed CNN-TEMMA improves emotion prediction performance compared to state-of-art models. Our approach obtains remarkable improvements on both arousal and valence in terms of concordance correlation coefficient (CCC) reaching 0.872 for arousal and 0.714 for valence.

Multimodal sentiment analysis aims to predict sentiments from multimodal signals such as audio, video, and text. Existing methods often rely on Pre-trained Language Models (PLMs) to extract semantic information from textual data, lacking an in-depth understanding of the logical relationships within the text modality . This article introduces the Multimodal PEAR (Preliminaries, quEstion, Answer, Reason) Chain-of-Thought (MM-PEAR-CoT) reasoning for multimodal sentiment analysis. Inspired by the human thought process when solving complex problems, the PEAR CoT prompt is first proposed to induce Large Language Models (LLMs) to generate text-based reasoning processes and zero-shot sentiment prediction results. However, text-based CoT reasoning is not always reliable and might contain irrational steps due to the hallucinations of LLMs . To address this, we further design the Cross-Modal Filtering and Fusion (CMFF) module. The filtering submodule utilizes audio and visual modalities to suppress irrational steps in the CoT, while the fusion submodule integrates high-level reasoning information and cross-modal complementary information in the process of semantic representation learning. Experimental results on two multimodal sentiment analysis benchmark datasets show that high-level reasoning information can help learn discriminative text representation, and cross-modal complementary information can avoid misleading by unreasonable steps in the CoT. MM-PEAR-CoT achieves the best results on both datasets, with improvements of 2.2% and 1.7% in binary classification accuracy on the CMU-MOSI and CMU-MOSEI datasets, respectively. To the best of our knowledge, this is the first study to apply CoT reasoning to multimodal sentiment analysis.

Major depressive disorder (MDD) is projected to become one of the leading mental disorders by 2030. While audiovisual cues have garnered significant attention in depression recognition research owing to their non-invasive acquisition and rich emotional expressiveness. However, conventional centralized training paradigms raise substantial privacy concerns for individuals with depression and are further hindered by data heterogeneity and label inconsistency across datasets. To overcome these challenges, a hybrid architecture, termed Federated Domain Adversarial with Attention Mechanism (FedDAAM), for privacy preserving multimodal depression assessment, is proposed. FedDAAM introduces a mechanism by differentiating discriminative features into depression-public and depression-private features. Specifically, to extract visual depression-private features from the AVEC2013 and AVEC2014 datasets, a local attention-aware (LAA) architecture is developed. For the depression-public features, action units (AUs), landmarks, head poses, and eye gazes features are adopted. In addition, to consider the transferability and performance of individual client, a dynamic parameter aggregation mechanism, termed FedDyA, is proposed. Extensive validations are performed on the AVEC2013, AVEC2014 and AVEC2017 databases, resulting in root mean square error (RMSE) and mean absolute error (MAE) of 8.61/6.78, 8.59/6.77, and 4.71/3.68, respectively. More importantly, to the best of our knowledge, this is the first study to borrow federated learning (FL) for multimodal depression assessment. The proposed framework offers a novel solution for privacy-aware, distributed clinical diagnosis of depression. Code will be available at: https://github.com/helang818/FedDAAM/.

In low-light environments, the presence of numerous small, dense, and occluded objects challenges the effectiveness of conventional object detection methods, failing to achieve desirable results. To address this, this paper proposes an efficient object detection network, YOLO_GD, which is designed for precise detection of targets in low-light scenarios. This algorithm, based on the foundational framework of YOLOv5s, implements a cross-layer feature fusion method founded on an information gathering and distribution mechanism. This method mitigates the issue of information loss during inter-layer feature exchange and, building on this, constructs a Bi-level routing spatial attention module to reduce computational redundancy caused by the self-attention mechanism, thereby enhancing the model’s detection accuracy for small objects. Furthermore, through the introduction of a novel deformable convolution, a cross-stage local feature fusion module is established, enabling the model to capture the complex features of input data more accurately and improve detection precision for dense objects. Lastly, the introduction of a probabilistic distance metric in the bounding box regression loss function enhances the network model’s generalization capability, further increasing detection accuracy in occluded scenarios. Experimental results on the ExDark dataset demonstrate that compared to YOLOv5, there is a 5.97% improvement in mean average precision (mAP), effectively enhancing object detection performance in low-light conditions.

This paper presents our solution for the Semi-Supervised Multimodal Emotion Recognition Challenge (MER2023-SEMI), addressing the issue of limited annotated data in emotion recognition. Recently, the self-training-based Semi-Supervised Learning~(SSL) method has demonstrated its effectiveness in various tasks, including emotion recognition. However, previous studies focused on reducing the confirmation bias of data without adequately considering the issue of data imbalance, which is of great importance in emotion recognition. Additionally, previous methods have primarily focused on unimodal tasks and have not considered the inherent multimodal information in emotion recognition tasks. We propose a simple yet effective semi-supervised multimodal emotion recognition method to address the above issues. We assume that the pseudo-labeled samples with consistent results across unimodal and multimodal classifiers have a more negligible confirmation bias. Based on this assumption, we suggest using a class-balanced strategy to select top-k high-confidence pseudo-labeled samples from each class. The proposed method is validated to be effective on the MER2023-SEMI Grand Challenge, with the weighted F1 score reaching 88.53% on the test set.

Multimodal Emotion Recognition (MER) aims to automatically identify and understand human emotional states by integrating information from various modalities. However, the scarcity of annotated multimodal data significantly hinders the advancement of this research field. This paper presents our solution for the MER-SEMI sub-challenge of MER 2024. First, to better adapt acoustic modality features for the MER task, we experimentally evaluate the contributions of different layers of the pre-trained speech model HuBERT in emotion recognition. Based on these observations, we perform Parameter-Efficient Fine-Tuning (PEFT) on the layers identified as most effective for emotion recognition tasks, thereby achieving optimal adaptation for emotion recognition with a minimal number of learnable parameters. Second, leveraging the strengths of the acoustic modality, we propose a feature alignment pre-training method. This approach uses large-scale unlabeled data to train a visual encoder, thereby promoting the semantic alignment of visual features within the acoustic feature space. Finally, using the adapted acoustic features, aligned visual features, and lexical features, we employ an attention mechanism for feature fusion. On the MER2024-SEMI test set, the proposed method achieves a weighted F1 score of 88.90%, ranking fourth among all participating teams, validating the effectiveness of our approach.