Neural evaluation metrics derived for numerous speech generation tasks have recently attracted great attention. In this paper, we propose SVSNet, the first end-to-end neural network model to assess the speaker voice similarity between converted speech and natural speech for voice conversion tasks. Unlike most neural evaluation metrics that use hand-crafted features, SVSNet directly takes the raw waveform as input to more completely utilize speech information for prediction. SVSNet consists of encoder, co-attention, distance calculation, and prediction modules and is trained in an end-to-end manner. The experimental results on the Voice Conversion Challenge 2018 and 2020 (VCC2018 and VCC2020) datasets show that SVSNet outperforms well-known baseline systems in the assessment of speaker similarity at the utterance and system levels.
25 September 2025 in IEEE Transactions on Audio, Speech and Language Processing
Multimodal speech emotion recognition (SER) has emerged as pivotal for improving human–machine interaction. Researchers are increasingly leveraging both speech and textual information obtained through automatic speech recognition (ASR) to comprehensively recognize emotional states from speakers. Although this approach reduces reliance on human-annotated text data, ASR errors possibly degrade emotion recognition performance. To address this challenge, in our previous work, we introduced two auxiliary tasks, namely, ASR error detection and ASR error correction, and we proposed a novel multimodal fusion (MF) method for learning modality-specific and modality-invariant representations across different modalities. Building on this foundation, in this paper, we introduce two additional training strategies. First, we propose an adversarial network to enhance the diversity of modality-specific representations. Second, we introduce a label-based contrastive learning strategy to better capture emotional features. We refer to our proposed method as M4SER and validate its superiority over state-of-the-art methods through extensive experiments using IEMOCAP and MELD datasets.
Purpose: This paper aims to investigate the efficiency and viability of online pair selection methods for preference score collection in Speech Quality Assessment (SQA). It addresses the challenge of high annotation costs in traditional pairwise comparisons and explores whether preference data collected in a sparse manner can successfully produce accurate rankings and support robust SQA model training. Design/methodology/approach: We propose an improved aggregation method to derive system-level scores from raw preference data. To equalize uncertainty across system pairs, we introduce a novel bias-based pair selection method. Furthermore, we develop a pseudo-labeling strategy that utilizes unlabeled utterances to infer missing preferences, thereby augmenting the training data and enhancing SQA model performance. Findings: Experiments demonstrate that the proposed bias-based pair selection method achieves the most robust performance. Additionally, the proposed pseudo-labeling approach effectively compensates for data sparsity, significantly improving the SQA model's performance by leveraging unlabeled datasets. Originality/value: This paper fulfills an identified need for cost-effective SQA data collection. It bridges the gap between active pair selection and deep learning-based SQA model training by introducing a novel bias-based selection mechanism and a pseudo-labeling strategy that leverages system-level scores to predict missing preferences.