1 January 2026 in IEEE Transactions on Audio, Speech and Language Processing
Significant progress has been made in the field of speech enhancement in recent years, and most speech enhancement models denoise in either the time-frequency or time-domain; these methods rely on large amounts of paired clean-noisy speech data and may not generalize well on unseen noise types. Meanwhile, noise-augmented self-supervised representations can see more unlabeled data, showing strong denoising and generalization capabilities. How to more effectively utilize self-supervised representations for speech enhancement remains an open and challenging problem. In this paper, we first propose SERV, a speech enhancement framework based on noise-augmented self-supervised representations and vocoders, where the self-supervised model learns the mapping from speech waveforms to representations. By combining contrastive and consistency losses, the model learns noise-robust speech representations from large-scale noisy speech data. Second, in the fine-tuning stage, we build a vocoder that maps the learned representations back to speech and fine-tune the entire model using paired clean and noisy speech data. Finally, we conduct extensive experiments to evaluate the effects of different layers of representations and various types of self-supervised models. The experimental results demonstrate that our proposed method achieves strong performance and generalization.