Bimodal emotion recognition based on audio and text is widely adopted in video-constrained real-world applications such as call centers and voice assistants. However, existing systems suffer from limited cross-domain generalization and monolingual bias. To address these limitations, a cross-lingual bimodal emotion recognition method is proposed, integrating Mamba-based temporal encoders for audio (Wav2Vec2.0) and text (Jina-v3) with a Transformer-based cross-modal fusion architecture (BiFormer). Three corpus-adaptive augmentation strategies are introduced: (1) Stacked Data Sampling, in which short utterances are concatenated to stabilize sequence length; (2) Label Smoothing Generation based on Large Language Model, where the Qwen3-4B model is prompted to detect subtle emotional cues missed by annotators, producing soft labels that reflect latent emotional co-occurrences; and (3) Text-to-Utterance Generation, in which emotionally labeled utterances are generated by ChatGPT-5 and synthesized into speech using the DIA-TTS model, enabling controlled creation of affective audio–text pairs without human annotation. BiFormer is trained jointly on the English Multimodal EmotionLines Dataset and the Russian Emotional Speech Dialogs corpus, enabling cross-lingual transfer without parallel data. Experimental results show that the optimal data augmentation strategy is corpus-dependent: Stacked Data Sampling achieves the best performance on short, noisy English utterances, while Label Smoothing Generation based on Large Language Model better captures nuanced emotional expressions in longer Russian utterances. Text-to-Utterance Generation does not yield a measurable gain due to current limitations in expressive speech synthesis. When combined, the two best performing strategies produce complementary improvements, establishing new state-of-the-art performance in both monolingual and cross-lingual settings.

The growing demand for personalized human–computer interaction calls for methods that jointly model emotional states and personality traits. However, large-scale multimodal corpora annotated for both tasks are still lacking. This challenge stems from integrating diverse, task-specific corpora with divergent modality informativeness and domain characteristics. To address it, we propose SSL-MEPR, a semi-supervised multi-task cross-domain learning framework for Multimodal Emotion and Personality Recognition, which enables cross-task knowledge transfer without jointly labeled data. SSL-MEPR employs a three-stage strategy, progressively integrating unimodal single-task, unimodal multi-task, and multimodal multi-task models. Key innovations include Graph Attention Fusion, task-specific query-based cross-attention, predict projectors, and guide banks, which enable robust fusion and effective use of semi-labeled data via a modified GradNorm method. Evaluated on MOSEI (emotion) and FIv2 (personality), SSL-MEPR achieves a mean Weighted Accuracy (mWACC) of 70.26 and a mean Accuracy (mACC) of 92.88 in single-task cross-domain settings, outperforming state-of-the-art methods. Multi-task learning reveals domain-induced misalignment in modality informativeness but still uncovers consistent psychological patterns: sadness correlates with lower personality trait scores, while happiness aligns with higher ones. This work establishes a new paradigm for extracting cross-task psychological knowledge from disjoint multimodal corpora, demonstrating that semi-supervised multi-task cross-domain learning can bridge annotation gaps while preserving theoretically grounded emotion–personality relationships.

Multi-task analysis of a human’s psychological profile enables a more holistic representation of the individual, which is particularly valuable in personalization systems, HR technologies, and human–Artificial Intelligence interaction. However, such studies have not been conducted to date due to the lack of datasets jointly annotated for both emotion and personality traits, rendering conventional multi-task learning infeasible. We propose a semi-supervised cross-domain learning method that effectively integrates two separately annotated corpora, CMU-MOSEI (for emotion recognition) and ChaLearn First Impressions v2 (FIv2) (for personality trait assessment), without requiring additional labeling. The experimental setup comprises two stages: first, independent single-task models are trained to extract domain-specific features and generate baseline predictions; second, a joint cross-domain model with cross-attention blocks fuses emotional and personality-related representations. Final predictions are obtained by averaging the outputs of the singletask and joint models, enhancing robustness. We compare pre-trained encoders (Jina-v3 and BGE-en) and contextual decoders (Transformer and Mamba), using a hybrid loss function that combines supervised and semi-supervised components with confidence-based pseudo-labeling. Experiments show that the best performance is achieved with the Jina-v3 encoder and the Mamba contextual model: mWACC = 62.52 % (Mean Weighted Accuracy Classification) and mMF1 = 61.03 % (Mean Weighted F1-Measure) on CMU-MOSEI (Multimodal Opinion Sentiment and Emotion Intensity) corpus; mACC = 88.80 % (Mean Accuracy) and mCCC = 25.44 % (Mean Concordation Correlation Coefficient) on FIv2. The model demonstrates stable knowledge transfer across tasks and outperforms current state-ofthe- art methods. Attention visualization via Grad-CAM confirms the interpretability of predictions. The proposed method enables the development of scalable text-based psychological profiling systems under realistic annotation scarcity. It is applicable in recruitment, adaptive learning platforms, personalized chatbots, and computational psychometrics where simultaneous consideration of emotional states and stable personality traits is essential.