Different subjects may express a specific expression in different ways due to inter-subject variabilities. In this work, besides training deep-learned facial expression feature (emotional feature), we also consider the influence of latent face identity feature such as the shape or appearance of face. We propose an identity and emotion joint learning approach with deep convolutional neural networks (CNNs) to enhance the performance of facial expression recognition (FER) tasks. First, we learn the emotion and identity features separately using two different CNNs with their corresponding training data. Second, we concatenate these two features together as a deep-learned Tandem Facial Expression (TFE) Feature and feed it to the subsequent fully connected layers to form a new model. Finally, we perform joint learning on the newly merged network using only the facial expression training data. Experimental results show that our proposed approach achieves 99.31% and 84.29% accuracy on the CK+ and the FER+ database, respectively, which outperforms the residual network baseline as well as many other state-of-the-art methods.

Background: Reduced or absence of the response to name (RTN) has been widely reported as an early specific indicator for autism spectrum disorder (ASD), while few studies have quantified the RTN of toddlers with ASD in an automatic way. The present study aims to apply a multimodal machine learning system (MMLS) in early screening for toddlers with ASD based on the RTN.Methods: A total of 125 toddlers were recruited, including ASD (n = 61), developmental delay (DD, n = 31), and typical developmental (TD, n = 33). Procedures of RTN were, respectively, performed by the evaluator and caregiver. Behavioral data were collected by eight-definition tripod-mounted cameras and coded by the MMLS. Response score, response time, and response duration time were accurately calculated to evaluate RTN.Results: Total accuracy of RTN scores rated by computers was 0.92. In both evaluator and caregiver procedures, toddlers with ASD had significant differences in response score, response time, and response duration time, compared to toddlers with DD and TD (all P-values < 0.05). The area under the curve (AUC) was 0.81 for the computer-rated results, and the AUC was 0.91 for the human-rated results. The accuracy in the identification of ASD based on the computer- and human-rated results was, respectively, 74.8 and 82.9%. There was a significant difference between the AUC of the human-rated results and computer-rated results (Z = 2.71, P-value = 0.007).Conclusion: The multimodal machine learning system can accurately quantify behaviors in RTN procedures and may effectively distinguish toddlers with ASD from the non-ASD group. This novel system may provide a low-cost approach to early screening and identifying toddlers with ASD. However, machine learning is not as accurate as a human observer, and the detection of a single symptom like RTN is not sufficient enough to detect ASD.

Electroencephalogram (EEG) emotion recognition plays an important role in human–computer interaction. An increasing number of algorithms for emotion recognition have been proposed recently. However, it is still challenging to make efficient use of emotional activity knowledge. In this paper, based on prior knowledge that emotion varies slowly across time, we propose a temporal-difference minimizing neural network (TDMNN) for EEG emotion recognition. We use maximum mean discrepancy (MMD) technology to evaluate the difference in EEG features across time and minimize the difference by a multibranch convolutional recurrent network. State-of-the-art performances are achieved using the proposed method on the SEED, SEED-IV, DEAP and DREAMER datasets, demonstrating the effectiveness of including prior knowledge in EEG emotion recognition.

Behavioral observation plays an essential role in the diagnosis of Autism Spectrum Disorder (ASD) by analyzing children’s atypical patterns in social activities (e.g., impaired social interaction, restricted interests, and repetitive behavior). To date, this process still heavily relies on the questionnaire survey, clinical observation, or retrospective video analysis, leading to high demand for professionals with massive labor costs. This paper proposes a standardized platform for stimulating, gathering, analyzing, modeling, and interpreting human behavioral data in the application of computer-aided ASD diagnosis. By a structured assessment process, the proposed system can automatically evaluate children’s multiple social interaction skills using the captured audio-visual data and provide the final diagnostic suggestions. We collect a multimodal behavioral database of 95 participants (71 children with ASD and 24 age-matched typical controls) in a real clinic environment, the Third Affiliated Hospital of Sun Yat-sen University, China. On the clinical database, our proposed computer-aided ASD diagnosis system obtains an accuracy of 88.42% for identifying ASD children with an average age of 24 months, representing a performance comparable to top-level human experts. As a unified and replicable solution, it has good potential to be promoted to less developed areas with limited high-quality medical resources.