19 March 2025 in IEEE Transactions on Audio, Speech and Language Processing
Neural network-based automatic Mean Opinion Score (MOS) prediction plays a pivotal role in assessing the perceptual quality of synthetic speech. Historically, this field has struggled with limited data resources, a problem not fully resolved by existing methodologies. On the one hand, self-supervised learning (SSL) models could enhance the representational power of the feature extractor by extensive pre-training on large datasets, but fail to address the data scarcity problem affecting the downstream decoder. On the other hand, methods such as data augmentation and the incorporation of individual judges' assessments could improve data utilization, but hard to ensure a satisfactory balance between the scale and quality of the additional dataset. To address these challenges of decoder inefficiency and suboptimal data utilization, we introduce a retrieval-augmented MOS prediction method with prior knowledge integration. This method employs a new approach to using SSL features by retrieving similar instances in the feature space to obtain scores, thereby boosting decoder performance. We also optimize dataset utilization by exploiting quality-related information from the dataset as prior knowledge. Furthermore, our framework includes a dynamic retrieval scope adaptation and a fusion network, enhancing both robustness and accuracy while minimizing the need for manual parameter tuning. Experimental results demonstrate that our retrieval approach, enhanced by prior knowledge, significantly outperforms existing state-of-the-art methods in the synthetic speech MOS prediction task across various scenarios. Our code and checkpoints are available at https://github.com/NKU-HLT/RAMP_MOS.git.
5 December 2025 in IEEE Transactions on Audio, Speech and Language Processing
Multimodal emotion recognition (MMER) is crucial for effective human-computer interaction, yet its performance degrades sharply when modality data are incomplete due to privacy constraints or data loss. Existing approaches often attempt to reconstruct missing information directly from raw, heterogeneous modalities, requiring complex cross-modal mappings that lead to low-quality reconstruction and hence less performance improvement. To address these challenges, we propose a progressive learning framework (ProLF) for missing-modality reconstruction and robust multimodal fusion. ProLF’s core innovation lies in its design of an incremental three-stage strategy, i.e., Alignment, Reconstruction, and Augmentation-which collectively mitigates modality heterogeneity, data incompleteness, and weak fused representations. In the Alignment stage, ProLF applies unsupervised distribution-level contrastive learning to reduce cross-modal discrepancies and establish a unified latent space. In the Reconstruction stage, a normalizing flow model efficiently transforms the aligned latent distributions to recover missing modalities with high accuracy. In the Augmentation stage, a mixture of modality experts performs deep fusion of the restored multimodal information, while supervised point-based contrastive learning enhances emotion-specific features and suppresses irrelevant semantics, substantially improving discriminability. Extensive experiments on multiple benchmark datasets validate the effectiveness of ProLF. On IEMOCAP and MSP-IMPROV, ProLF achieves average performance gains of 1.5%–2%, and on CMU-MOSI, improvements of 1%–3%. These results confirm that ProLF provides robust and consistent benefits under both missing and complete modality conditions.