Emotion is a complex physiological phenomenon, and a single modality may be insufficient for accurately determining human emotional states. This paper proposes an end-to-end multimodal emotion recognition method based on facial expressions and non-contact physiological signals. Facial expression features and remote photoplethysmography (rPPG) signals are extracted from facial video data, and a transformer-based cross-modal attention mechanism (TCMA) is used to learn the correlation between the two modalities. The results show that the accuracy of emotion recognition can be slightly improved by combining facial expressions with accurate rPPG signals. The performance is further improved with the use of TCMA, for which the binary classification accuracy of valence and arousal is 91.11% and 90.00%, respectively. Additionally, when experiments are conducted using the whole dataset, an increased accuracy of 7.31% and 4.23% for the binary classification of valence and arousal, and an improved accuracy of 5.36% for the four classifications of valence-arousal are achieved when TCMA is used in modal fusion, compared to using only facial expression modality, which fully demonstrates the effectiveness and robustness of TCMA. This method makes it possible to realize multimodal emotion recognition of facial expressions and contactless physiological signals in reality.
1 December 2016 in Australasian Physics & Engineering Sciences in Medicine
The surface tension of liquid carbon dioxide has practical significance in material science, environmental science and chemical engineering separation processes as well as in the secondary or tertiary recovery of petroleum. The surface tension of liquid carbon dioxide is estimated by semi-empirical and statistical formulae and the results are compared and analysed with experimental data. It is shown that the methods proposed by Brock, Hakim, Miqueua and Zuo are better methods for estimating the surface tension of liquid carbon dioxide and have relatively less percentage deviation from experimental data.
21 June 2023 in American Journal of Otolaryngology