Audio-visual emotion recognition is the research of identifying human emotional states by combining the audio modality and the visual modality simultaneously, which plays an important role in intelligent human-machine interactions. With the help of deep learning, previous works have made great progress for audio-visual emotion recognition. However, these deep learning methods often require a large amount of data for training. In reality, data acquisition is difficult and expensive, especially for the multimodal data with different modalities. As a result, the training data may be in the low-data regime, which cannot be effectively used for deep learning. In addition, class imbalance may occur in the emotional data, which can further degrade the performance of audio-visual emotion recognition. To address these problems, we propose an efficient data augmentation framework by designing a multimodal conditional generative adversarial network (GAN) for audio-visual emotion recognition. Specifically, we design generators and discriminators for audio and visual modalities. The category information is used as their shared input to make sure our GAN can generate fake data of different categories. In addition, the high dependence between the audio modality and the visual modality in the generated multimodal data is modeled based on Hirschfeld-Gebelein-Rényi (HGR) maximal correlation. In this way, we relate different modalities in the generated data to approximate the real data. Then, the generated data are used to augment our data manifold. We further apply our approach to deal with the problem of class imbalance. To the best of our knowledge, this is the first work to propose a data augmentation strategy with a multimodal conditional GAN for audio-visual emotion recognition. We conduct a series of experiments on three public multimodal datasets, including eNTERFACE’05, RAVDESS, and CMEW. The results indicate that our multimodal conditional GAN has high effectiveness for data augmentation of audio-visual emotion recognition.

Research on traumatic events often emphasizes the importance of posttraumatic growth (PTG) and resilience, yet few studies have explored their trends and their relationship throughout the progression of traumatic events. This paper explores the longitudinal relationship between resilience and PTG, as well as the role of job burnout in this relationship, among frontline healthcare workers during the COVID‐19 pandemic, who have been exposed to high‐risk work environments over extraordinarily long workdays. In Study 1, 134 Chinese frontline healthcare workers completed a three‐wave survey (Time 1, Time 2, and Time 3) in February–May 2020. In Study 2, 401 frontline healthcare workers completed a cross‐sectional survey. The cross‐lagged analysis suggested that resilience at Time 1 positively predicted PTG at Time 2, which in turn positively predicted resilience at Time 3. PTG at Time 1 also positively predicted resilience at Time 2 (Study 1). However, job burnout was negatively related to both resilience and PTG; in particular, emotional exhaustion moderated the link between PTG and resilience (Study 2). Our findings support a cycle of reinforcement between resilience and PTG over time. The positive effect of PTG on resilience, however, is undermined by emotional exhaustion. Implications for future intervention research and workplace support are discussed.

Background: Heart rate variability (HRV) provides essential mental health information for clinical diagnosis, telemedicine, preventive medicine, and public health; however, the lack of a convenient detection method limits its potential. Objective: This study aims to investigate the feasibility and credibility of smartphone photoplethysmogram (PPG)-based HRV analysis for mental well-being and health assessment. Methods: Data were collected from 93 students and university employees in Shenzhen, China. Forty-six percent were male, and the average age was 23.71 years (σ = 4.33). An app recorded a 4-min video of their fingertips and converted the frames into five HRV measures, including the root mean square of successive differences (rMSSD), standard deviation of the normal-to-normal (NN) intervals (SDNN), percentage of successive NN intervals differing by ≥50 ms (pNN50), log high-frequency (HF) HRV, and log low-frequency (LF) HRV. Results: The data verify the positive relationship between mental well-being and HRV measures. Participants with higher Satisfaction With Life Scale (SWLS) scores have a higher rMSSD ( p = 0.047), SDNN ( p = 0.009), log HF ( p = 0.02), and log LF ( p = 0.003). Participants who suffer from depression have lower log HF ( p = 0.048) and log LF ( p = 0.02). Participants in the high-anxiety group have lower pNN50 ( p = 0.04) and log HF ( p = 0.03). Conclusions: The results of this study validate the feasibility of using the smartphone PPG by demonstrating similar results to previous findings. Our data also support the theorized positive link between mental health and HRV.