Emotion recognition is becoming increasingly important for accurately understanding and responding to user emotions, driven by the rapid proliferation of non-face-to-face environments and advancements in conversational AI technologies. Existing studies on multimodal emotion recognition, which utilize text and speech, have aimed to improve performance by integrating the information from both modalities. However, these approaches have faced limitations such as restricted information exchange and the omission of critical cues. To address these challenges, this study proposes a Hybrid Multimodal Transformer, which combines Intermediate Layer Fusion and Last Fusion. Text features are extracted using KoELECTRA, while speech features are extracted using HuBERT. These features are processed through a transformer encoder, and Dual Cross Modal Attention is applied to enhance the interaction between text and speech. Finally, the predicted results from each modality are aggregated using an average ensemble method to recognize the final emotion. The experimental results indicate that the proposed model achieves superior emotion recognition performance compared to existing models, demonstrating significant progress in improving both the accuracy and reliability of emotion recognition. In the future, incorporating additional modalities, such as facial expression recognition, is expected to further strengthen multimodal emotion recognition capabilities and open new possibilities for application across diverse fields.

With the advancement of human-computer interaction, the role of emotion recognition has become increasingly significant. Emotion recognition technology provides practical benefits across various industries, including user experience enhancement, education, and organizational productivity. For instance, in educational settings, it enables real-time understanding of students’ emotional states, facilitating tailored feedback. In workplaces, monitoring employees’ emotions can contribute to improved job performance and satisfaction. Recently, emotion recognition has also gained attention in media applications such as automated movie dubbing, where it enhances the naturalness of dubbed performances by synchronizing emotional expression in both audio and visuals. Consequently, multimodal emotion recognition research, which integrates text, speech, and video data, has gained momentum in diverse fields. In this study, we propose an emotion recognition approach that combines text and speech data, specifically incorporating the characteristics of the Korean language. For text data, we utilize KoELECTRA to generate embeddings, and for speech data, we extract features using HuBERT embeddings. The proposed multimodal transformer model processes text and speech data independently, subsequently learning interactions between the two modalities through a Cross-Modal Attention mechanism. This approach effectively combines complementary information from text and speech, enhancing the accuracy of emotion recognition. Our experimental results demonstrate that the proposed model surpasses single-modality models, achieving a high accuracy of 77.01% and an F1-Score of 0.7703 in emotion classification. This study contributes to the advancement of emotion recognition technology by integrating diverse language and modality data, suggesting the potential for further improvements through the inclusion of additional modalities in future work.

With the development of human-computer interaction, the importance of emotion recognition is increasing. Emotion recognition technology provides practical benefits in various industries such as improving user experience, education, and organizational productivity. In education, emotion recognition can enable real-time understanding of students' emotional states and provide tailored feedback, and in the workplace, by monitoring employees' emotional states, it can improve work performance and job satisfaction. Therefore, multimodal-based research that combines text, speech, and video data for emotion recognition is being conducted in various industries. In this study, we propose an emotion recognition method that combines text and speech data, reflecting the characteristics of the Korean language. For text, KoELECTRA is used to perform embedding, and for speech, MFCC and pitch analysis are used to extract features. Finally, a multimodal transformer model is proposed that combines these two data types to perform emotion recognition. The multimodal transformer model processes text and speech data separately, and then learns the interaction between the two modalities through a Cross-Modal Attention mechanism. Through this, the complementary information from text and speech is effectively combined, improving emotion recognition performance. Experimental results show that the proposed model outperforms single-modality models, achieving a high accuracy of 73.13% and an F1-Score of 0.7344 in emotion classification. This study contributes to the advancement of emotion recognition technology by combining various language and modality data, and higher performance can be expected through the integration of additional modalities in the future.

Computer communication via text messaging or Social Networking Services (SNS) has become increasingly popular. At this time, many studies are being conducted to analyze user information or opinions and recognize emotions by using a large amount of data. Currently, the methods for the emotion recognition of dialogues requires an analysis of emotion keywords or vocabulary, and dialogue data are mostly classified as a single emotion. Recently, datasets classified as multiple emotions have emerged, but most of them are composed of English datasets. For accurate emotion recognition, a method for recognizing various emotions in one sentence is required. In addition, multi-emotion recognition research in Korean dialogue datasets is also needed. Since dialogues are exchanges between speakers. One’s feelings may be changed by the words of others, and feelings, once generated, may last for a long period of time. Emotions are expressed not only through vocabulary, but also indirectly through dialogues. In order to improve the performance of emotion recognition, it is necessary to analyze Emotional Association in Dialogues (EAD) to effectively reflect various factors that induce emotions. Therefore, in this paper, we propose a more accurate emotion recognition method to overcome the limitations of single emotion recognition. We implement Intrinsic Emotion Recognition (IER) to understand the meaning of dialogue and recognize complex emotions. In addition, conversations are classified according to their characteristics, and the correlation between IER is analyzed to derive Emotional Association in Dialogues (EAD) and apply them. To verify the usefulness of the proposed technique, IER applied with EAD is tested and evaluated. This evaluation determined that Micro-F1 of the proposed method exhibited the best performance, with 74.8% accuracy. Using IER to assess the EAD proposed in this paper can improve the accuracy and performance of emotion recognition in dialogues.

Due to the recent increase in non-face-to-face services due to COVID-19, the number of users communicating through messengers or SNS (social networking service) is increasing. As a large amount of data is generated by users, research on recognizing emotions by analyzing user information or opinions is being actively conducted. Conversation data such as SNS is freely created by users, so there is no set format. Due to these characteristics, it is difficult to analyze using AI (artificial intelligence), which leads to a decrease in the performance of the emotion recognition technique. Therefore, a processing method suitable for the characteristics of unstructured data is required. Among the unstructured data, most emotion recognition in Korean conversation recognizes a single emotion by analyzing emotion keywords or vocabulary. However, since multiple emotions exist complexly in a single sentence, research on multilabel emotion recognition is needed. Therefore, in this paper, the characteristics of unstructured conversation data are considered and processed for more accurate emotion recognition. In addition, we propose a multilabel emotion recognition technique that understands the meaning of dialogue and recognizes inherent and complex emotions. A deep learning model was compared and tested as a method to verify the usefulness of the proposed technique. As a result, performance was improved when it was processed in consideration of the characteristics of unstructured conversation data. Also, when the attention model was used, accuracy showed the best performance with 65.9%. The proposed technique can contribute to improving the accuracy and performance of conversational emotion recognition.

This paper presents a novel dual-stream framework for multimodal emotion recognition, engineered to address the varying complexity inherent in emotional expressions. The proposed architecture uniquely integrates a graph embedding as an auxiliary modality to explicitly model temporal correlations between utterances, and processes features through two complementary sub-models operating in parallel: a Cross-Attention Transformer Mixture-of-Experts (MoE) model and a Sum-Product Linear MoE model. The former deciphers nuanced and ambiguous emotions, such as ‘fear’ and ‘sadness’, by leveraging a deep cross-attention mechanism to model intricate, bidirectional dependencies between textual and acoustic features. The latter is a lightweight model optimized to efficiently recognize clear and intuitive emotions, like ‘anger’ and ‘happiness’, through simple element-wise operations. The final prediction is derived from an average ensemble of logits from both sub-models, ensuring a robust and balanced classification. Evaluated on the Korean AI-Hub and English IEMOCAP datasets, the framework achieves state-of-the-art accuracy of 0.8071 and demonstrates excellent cross-lingual generalization with an accuracy of 0.7823. Empirical results validate the complementary design, confirming that the specialized models synergistically enhance overall performance.