Multi-modal continuous affect recognition is becoming an increasingly attractive research topic in affective computing. Previous works mainly focus on modeling the temporal dependency within a sensor modality. To improve system performance most of them adopt early or late fusion in multi-modal continuous affect recognition. However, early fusion suffers from the curse of dimensionality, and late fusion ignores the complementarity and redundancy between multiple streams from different modalities. In this paper, we first introduce the transformer encoder with a self-attention mechanism to temporal dependencies affective state time series and propose a Convolutional Neural Network-Transformer Encoder (CNN-TE) framework for single modality affect recognition. To effectively consider the complementarity and redundancy between multiple streams from different modalities we propose a Transformer Encoder with Multi-modal Multi-head Attention (TEMMA) module designed to progressively refine the inter-modality interactions and intra-modality temporal dependencies simultaneously. We propose a CNN-TEMMA framework for multi-modal affect recognition, which consists of (i)~a 1D CNN based input embedding sub-network to extract local temporal features via causal convolution; (ii)~a TEMMA sub-network to model the multi-modal interactions and temporal dynamics with spatial and temporal attention, to learn high-level representations for each modality; (iii)~an inference sub-network with fully connected layers to estimate the affective state, using as inputs the learned multi-modal representations. The whole framework is trained in an end-to-end fashion. We demonstrate the effectiveness of our proposed framework on the AVEC2016 dataset. Our experimental results show that the proposed CNN-TEMMA improves emotion prediction performance compared to state-of-art models. Our approach obtains remarkable improvements on both arousal and valence in terms of concordance correlation coefficient (CCC) reaching 0.872 for arousal and 0.714 for valence.

Depression causes mood disorder with noticeable problems in day-to-day activities. Current methods of assessing depression depend almost entirely on clinical interviews or questionnaires. They lack systematic and efficient ways of incorporating behavioral observations that are strong indicators of psychological disorder. To help clinicians effectively and efficiently diagnose depression severity, automated systems, using objective and quantifiable data for depression assessment, are being developed. This paper presents a framework towards estimating a clinical depression-specific score, namely the Beck Depression Inventory (BDI) score, based on the analysis of facial expressions features. To extract facial dynamic features, we propose a novel dynamic feature descriptor denoted as Median Robust Local Binary Patterns from Three Orthogonal Planes (MRLBP-TOP), which can capture both microstructure and macrostructure of facial appearance and dynamics. To aggregate the MRLBP-TOP over an image sequence, we propose a variant to the Fisher Vector (FV) encoding scheme, denoted as Dirichlet Process Fisher Vector (DPFV). DPFV adopts Dirichlet process Gaussian mixture models (DPGMM) to automatically learn the number of GMM mixtures and model parameters. Experimental results on the AVEC2013 and AVEC2014 depression databases have demonstrated the effectiveness of the proposed method.

At present, although great progress has been made in automatic depression assessment, most of the recent works only concern the audio and video paralinguistic information, rather than the linguistic information from the spoken content. In this work, we argue that beside developing good audio and video features, to build reliable depression detection systems, text-based content features are also of importance to analyse depression-related textual indicators. Furthermore, to improve the performance of automatic depression assessment systems, powerful models, capable of modelling the characteristics of depression embedded in the audio, visual and text descriptors, are also required. This paper proposes new text and video features and hybridizes deep and shallow models for depression estimation and classification from audio, video and text descriptors. The proposed hybrid framework consists of three main parts: 1) A Deep Convolutional Neural Network (DCNN) and Deep Neural Network (DNN) based audio-visual multi-modal depression recognition model for estimating the Patient Health Questionnaire depression scale (PHQ-8); 2) A Paragraph Vector (PV) and Support Vector Machine (SVM) based model for inferring the physical and mental conditions of the individual from the transcripts of the interview; 3) A Random Forest (RF) model for depression classification from the estimated PHQ-8 score and the inferred conditions of the individual. In the PV-SVM model, PV embedding is used to obtain fixed-length feature vectors from transcripts of the answers to the questions associated with psychoanalytic aspects of depression, which are subsequently fed into the SVM classifiers for detecting the presence/absence of the considered psychoanalytic symptoms. To our best knowledge, this approach is the first attempt to apply PV for depression analysis. Besides, we propose a new visual descriptor - Histogram of Displacement Range (HDR) to characterize the displacement and velocity of the facial landmarks in the video segment. Experiments have been carried out on the Audio Visual Emotion Challenge (AVEC2016) depression dataset, they demonstrate that: 1) The proposed hybrid framework effectively improves the accuracies of both depression estimation and depression classification, with an average F1 measure up to 0.746, which is higher than the best result (0.724) of the depression sub-challenge of AVEC2016. 2) HDR obtains better depression recognition performance than Bag-of-Words (BoW) and Motion History Histogram (MHH) features.

Depression disorder has become one of the major psychological diseases endangering human health. Researcher in the affective computing community is supporting the development of reliable depression severity estimation system, from multiple modalities (speech, face, text), to assist doctors in their diagnosis. However, the limited amount of annotated data has become the main bottleneck restricting the study on depression screening, especially when deep learning models are used. To alleviate this issue, in this work we propose to use Deep Convolutional Generative Adversarial Network (DCGAN) for features augmentation to improve depression severity estimation from speech. To the best of our knowledge, this approach is the first attempt to apply the Generative Adversarial Network for depression severity estimation from speech. Besides, to measure the quality of the augmented features, we propose three different measurement criteria, characterizing the spatial, frequency and representation learning of the augmented features. Finally, the augmented features are used to train depression estimation models. Experiments are carried out on speech signals from the Audio Visual Emotion Challenge (AVEC2016) depression dataset, and the relationship between the model performance and data size is explored. Our experimental results show that: 1) The combination of the three proposed evaluation criteria can effectively and comprehensively evaluate the quality of the augmented features. 2) When increasing the size of the augmented data, the performance of depression severity estimation gradually improves and the model converges to a certain stable state. 3) The proposed DCGAN based data augmentation approach effectively improves the performance of depression severity estimation, with the root mean square error (RMSE) reduced to 5.520 and mean absolute error (MAE) reduced to 4.634, which is better than most of the state of the art results on AVEC 2016.

Electroencephalography (EEG) is widely used for mental stress classification, but effective feature extraction and transfer across subjects remain challenging due to its variability. In this paper, a novel deep neural network combining convolutional neural network (CNN) and adversarial theory, named symmetric deep convolutional adversarial network (SDCAN), is proposed for stress classification based on EEG. The adversarial inference is introduced to automatically capture invariant and discriminative features from raw EEG, which aims to improve the classification accuracy and generalization ability across subjects. Experiments were conducted with 22 human subjects, where each participant’s stress was induced by the Trier Social Stress Test paradigm while EEG was collected. Stress states were then calibrated into four or five stages according to the changing trend of salivary cortisol concentration. The results show that the proposed network achieves improved accuracies of 87.62% and 81.45% on the classification of four and five stages, respectively, compared to conventional CNN methods. Euclidean space data alignment approach (EA) was applied and the improved generalization ability of EA-SDCAN across subjects was also validated via the leave-one-subject-out-cross-validation, with the accuracies of four and five stages being 60.52% and 48.17%, respectively. These findings indicate that the proposed SDCAN network is more feasible and effective for classifying the stages of mental stress based on EEG compared with other conventional methods.

Onboard real-time object detection in remote sensing images is a crucial but challenging task in this computation-constrained scenario. This task not only requires the algorithm to yield excellent performance but also requests limited time and space complexity of the algorithm. However, previous convolutional neural networks (CNN) based object detectors for remote sensing images suffer from heavy computational cost, which hinders them from being deployed on satellites. Moreover, an onboard detector is desired to detect objects at vastly different scales. To address these issues, we proposed a lightweight one-stage multi-scale feature fusion detector called MSF-SNET for onboard real-time object detection of remote sensing images. Using lightweight SNET as the backbone network reduces the number of parameters and computational complexity. To strengthen the detection performance of small objects, three low-level features are extracted from the three stages of SNET respectively. In the detection part, another three convolutional layers are designed to further extract deep features with rich semantic information for large-scale object detection. To improve detection accuracy, the deep features and low-level features are fused to enhance the feature representation. Extensive experiments and comprehensive evaluations on the openly available NWPU VHR-10 dataset and DIOR dataset are conducted to evaluate the proposed method. Compared with other state-of-art detectors, the proposed detection framework has fewer parameters and calculations, while maintaining consistent accuracy.