1 June 2024 in EURASIP Journal on Audio, Speech, and Music Processing
Limited data availability remains a significant challenge for Whisper’s low-resource speech recognition performance, falling short of practical application requirements. While previous studies have successfully reduced the recognition error rates of target language speech through fine-tuning, a comprehensive exploration and analysis of Whisper’s fine-tuning capabilities and the advantages and disadvantages of various fine-tuning strategies are still lacking. This paper aims to fill this gap by conducting comprehensive experimental exploration for Whisper’s low-resource speech recognition performance using five fine-tuning strategies with limited supervised data from seven low-resource languages. The results and analysis demonstrate that all fine-tuning strategies explored in this paper significantly enhance Whisper’s performance. However, different strategies vary in their suitability and practical effectiveness, highlighting the need for careful selection based on specific use cases and resources available.
3 January 2025 in IEEE Signal Processing Letters
Parameter-efficient fine-tuning of pre-trained multilingual speech models can significantly enhance the speech recognition performance of target languages. However, traditional parameter-efficient fine-tuning methods, such as adapter tuning, often face challenges related to random initialization. This can lead to suboptimal performance when adapting to languages with limited resources. To address this issue, this paper introduces TAML-Adapter, which utilizes the Task-Agnostic Meta-Learning algorithm to initialize the parameters of the adapters before fine-tuning in target low-resource languages. Comprehensive experiments conducted on the Common Voice and Fleurs datasets highlight the superior performance of TAML-Adapter in five languages with limited resources. In addition, the TAML-Adapter demonstrates superior generalizability and extensibility compared to similar competing methods.
5 May 2025 in Neural Processing Letters
Multilingual automatic speech recognition represents a crucial research direction in tackling the challenges associated with low-resource scenarios. To effectively incorporate language-specific information in the joint training of a model across multiple languages and leverage linguistic similarities to enhance performance on the target low-resource language, this paper introduces a language similarity evaluation approach based on forced decoding. Specifically, when the target language is specified, the speeches of the source language are decoded into transcription in the target language, and the normalized posterior is utilized as the foundation for evaluating language similarity. Comprehensive experiments and analyses conducted on six low-resource languages reveal that the proposed approach achieves an average word error rate relative reduction of 21.74, 7.68, and 3.45% compared to three widely used benchmark methods, respectively, thereby validating the effectiveness of our approach.
21 October 2025 in Signal, Image and Video Processing