The style transfer task in Text-to-Speech (TTS) refers to the process of transferring style information into text content to generate corresponding speech with a specific style. However, most existing style transfer approaches are either based on fixed emotional labels or reference speech clips, which cannot achieve flexible style transfer. Recently, some methods have adopted text descriptions to guide style transfer. In this paper, we propose a more flexible multi-modal and style controllable TTS framework named MM-TTS. It can utilize any modality as the prompt in unified multi-modal prompt space, including reference speech, emotional facial images, and text descriptions, to control the style of the generated speech in a system. The challenges of modeling such a multi-modal style controllable TTS mainly lie in two aspects: 1) aligning the multi-modal information into a unified style space to enable the input of arbitrary modality as the style prompt in a single system, and 2) efficiently transferring the unified style representation into the given text content, thereby empowering the ability to generate prompt style-related voice. To address these problems, we propose an aligned multi-modal prompt encoder that embeds different modalities into a unified style space, supporting style transfer for different modalities. Additionally, we present a new adaptive style transfer method named Style Adaptive Convolutions (SAConv) to achieve a better style representation. Furthermore, we design a Rectified Flow based Refiner to solve the problem of over-smoothing Mel-spectrogram and generate audio of higher fidelity. Since there is no public dataset for multi-modal TTS, we construct a dataset named MEAD-TTS, which is related to the field of expressive talking head. Our experiments on the MEAD-TTS dataset and out-of-domain datasets demonstrate that MM-TTS can achieve satisfactory results based on multi-modal prompts. The audio samples and constructed dataset are available at https://multimodal-tts.github.io.

Based on respiratory sound production mechanisms, we study the relationship between airflow characteristics in the bronchi and sound pressure spectrum curves to implement an end-to-end respiratory sound classification system with a feature-band attention module. First, we analyse fluid-solid coupling simulations of the bronchi and execute acoustic simulations to obtain the spectrum curves of the bronchi at the sound pressure level. Then, based on the spectrum characteristics of the bronchi, we propose an attention strategy to refine the acoustic features with adaptive weights. In addition, we introduce a feature-band attention module to ResNet-based networks with a squeeze-and-excitation block. Finally, we perform experiments on the ICBHI public database to classify respiratory sounds into one of four classes: normal, wheezes, crackles, and both (wheezes and crackles). The results show that our proposed system exhibits superior performance compared with the baseline system. This type of feature learning strategy is useful for exploring the distinct characteristics of different types of respiratory sounds.

A typical speaker recognition system often involves two modules: a feature extractor front-end and a speaker identification back-end. Despite the superior performance that deep neural networks have achieved for the front-end, their success benefits from the availability of large-scale, correctly labeled datasets. While label noise is unavoidable in speaker recognition datasets, both the front-end and back-end are affected by label noise, which degrades speaker recognition performance. In this paper, we first conduct comprehensive experiments to help improve our understanding of the effects of label noise on both the front-end and back-end. Then, we propose a simple yet effective training paradigm and loss correction method to handle label noise in the front-end. We combine our proposed method with the recently proposed Bayesian estimation of PLDA for noisy labels, and the whole system shows strong robustness to label noise. Furthermore, we show two practical applications of the improved system: one application corrects noisy labels based on an utterance’s chunk-level predictions, and the other algorithmically filters out high-confidence noisy samples within a dataset. By applying the second application to the NIST SRE04–10 dataset and verifying filtered utterances by human validation, we identify that approximately 1% of the NIST SRE04–10 dataset is made up of label errors.

Supervised deep learning has achieved tremendous success in speaker verification. However, deep speaker models tend to overfit noisy labels when they are present in the speaker datasets. To mitigate the detrimental effects of noisy labels, in this paper, we propose a novel Label Ensembling and Sample Selection framework. Firstly, we select labels with high confidence rankings as clean samples. Additionally, we use predictions from different epochs during training to smoothly correct the noisy labels. Our method does not require staged training and achieves integration of learning from noisy labels, selecting clean labels, and correcting noisy labels. A significant number of experimental results demonstrate the robustness of our method under noisy labels. Even when the training data contains 50% noisy labels, our method can mitigate an average of 86.54% of the performance degradation compared to the standard training method. Furthermore, further ablation experiments and analysis validate the effectiveness of High Confidence Ranking for sample selection and the correctness of Label Ensembling for noisy label correction.