The clinical diagnosis and treatment of motor dysarthria in post-stroke patients is often subjective and neglects the impact of psychological and emotional disorders on disease progression. This study aims to analyze the correlation among emotional expression, psychological state, facial expression, and dysarthria disease severity and is dedicated to the construction of a dysarthria prediction model. We first designed THE-POSSD, a novel Chinese multimodal emotional pathology expression database, which collected acoustic, glottal, and facial data under emotional stimuli from patients at different disease stages and healthy controls. Emotional speech was labeled for intelligibility scores, emotion types, and discrete dimensional space. Then, their correlation with disease development was investigated and analyzed. A total of 154 significant correlation features were extracted for analysis. To mitigate the limitations of subjective clinical scale diagnosis and account for psychological and emotional factors, this study introduced the grey correlation theory and constructed a dysarthria prediction model based on the grey relational analysis-deep belief network (GRA-DBN). Principal Component Analysis and Variance Inflation Factor were employed to optimize GRA-DBN model. Both proposed models achieved a high prediction accuracy, with an adjusted R² value of 0.85 for GRA-DBN and 0.92 for optimised model. This study fills the gap in the international multimodal emotional pathological expression dataset and provides a comprehensive framework for analyzing the association between mental state, emotional expression, and the degree of dysarthria. Furthermore, the incorporation of key multimodal features into the predictive model highlights its potential to enhance the precision of clinical diagnostic processes significantly.
There are still challenges in speech emotion recognition, as the representation capability of single-modal information is limited, there are difficulties in capturing continuous emotional transitions in discrete emotion annotations, and the issues of modal structural differences and cross-sample alignment in multimodal fusion methods persist. To address these, this study undertakes work from both data and model perspectives. For data, a Chinese multimodal database STEM-E2VA was constructed, synchronously collecting four modalities of data: articulatory kinematics, acoustics, glottal signals, and videos. This covers seven discrete emotion categories and employs PAD continuous annotation. By integrating discrete and continuous dimensional annotations, it better represents the distinction between strong and weak emotions under the same discrete emotion label. Concurrently, to process the biases in PAD annotations, we employed the SCL-90 psychological questionnaire to analyze annotators’ cognitive and emotional perceptions, thereby ensuring data reliability. For model, this paper proposes a multimodal supervised contrastive fusion network incorporating PAD perception. It employs a PAD-enhanced hybrid contrastive loss function to optimize intra-model and inter-modal feature alignment. Utilizing a cross-attention mechanism combined with a GRU–Transformer network for temporal feature extraction, it achieves deep fusion of multimodal information, reducing inter-modal discrepancies and cross-class confusion. Experiments demonstrate that the proposed method achieves 85.47% accuracy in discrete sentiment recognition on STEM-E2VA, with a substantial reduction in RMSE for PAD dimension prediction. It also exhibits excellent generalization capability on IEMOCAP, providing a novel framework for integrating discrete and continuous sentiment representations.
With the rise of AI-generated content (AIGC), generating perceptually natural and feeling-aligned music from multimodal inputs has become a central challenge. Existing approaches often rely on explicit emotion labels that require costly annotation, underscoring the need for more flexible feeling-aligned methods. To support multimodal music generation, we construct ArtiCaps, a pseudo feeling-aligned image-music-text dataset created by semantically matching descriptions from ArtEmis and MusicCaps. We further propose Art2Music, a lightweight cross-modal framework that synthesizes music from artistic images and user comments. In the first stage, images and text are encoded with OpenCLIP and fused using a gated residual module; the fused representation is decoded by a bidirectional LSTM into Mel-spectrograms with a frequency-weighted L1 loss to enhance high-frequency fidelity. In the second stage, a fine-tuned HiFi-GAN vocoder reconstructs high-quality audio waveforms. Experiments on ArtiCaps show clear improvements in Mel-Cepstral Distortion, Frechet Audio Distance, Log-Spectral Distance, and cosine similarity. A small LLM-based rating study further verifies consistent cross-modal feeling alignment and offers interpretable explanations of matches and mismatches across modalities. These results demonstrate improved perceptual naturalness, spectral fidelity, and semantic consistency. Art2Music also maintains robust performance with only 50k training samples, providing a scalable solution for feeling-aligned creative audio generation in interactive art, personalized soundscapes, and digital art exhibitions.