Wupeng Wang

Wupeng Wang

Auto-generated profile, based on publicly available information

Wupeng Wang

School of Intelligence Science and Technology, Nanjing University, Suzhou, China

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
1 May 2020
Verified
Conference Paper
Closed access
WWWupeng Wang
Wupeng Wang
Xing ChaoXing Chao
Xing Chao
DWDong Wang
Dong Wang
XCXiao Chen
Xiao Chen
FSFengyu Sun
Fengyu Sun
Most existing audio-visual speech enhancement (AVSE) methods work well in conditions with strong noise, however when applied to conditions with a medium SNR, serious performance degradations are often observed. These degradations can be partly attributed to the feature-fusion(early fusion etc.) architecture that tightly couples the audio information that is very strong and the visual information that is relatively weak. In this paper, we present a safe AVSE approach that can make the visual stream contribute to audio speech enhancment(ASE) safely in conditions of various SNRs by late fusion.The key novelty is two-fold: Firstly, we define power binary masks (PBMs) as a rough representation of speech signals. This rough representation admits the weakness of the visual information and so can be easily predicted from the visual stream. Secondly, we design a posterior augmentation architecture that integrate the visual-derived PBMs to the audio-derived masks via a gating network. By this architecture, the entire performance is lower-bounded by the audio-based component. Our experiments on the Grid dataset demonstrated that this new approach consistently outperforms the audio-based system in all noise conditions, confirming that it is a safe way to incorporate visual knowledge in speech enhancement.

3 April 2024 in IEEE Signal Processing Letters
Verified
Journal Article
Open access
JLJingru Lin
Jingru Lin
MGMeng Ge
Meng Ge
WWWupeng Wang
Wupeng Wang
HLHaizhou Li
Haizhou Li
MFMengling Feng
Mengling Feng
Self-supervised pre-trained speech models were shown effective for various downstream speech processing tasks. Since they are mainly pre-trained to map input speech to pseudo-labels, the resulting representations are only effective for the type of pre-train data used, either clean or mixture speech. With the idea of selective auditory attention, we propose a novel pre-training solution called Selective-HuBERT, or SHuBERT, which learns the selective extraction of target speech representations from either clean or mixture speech. Specifically, SHuBERT is trained to predict pseudo labels of a target speaker, conditioned on an enrolment speech from the target speaker. By doing so, SHuBERT is expected to selectively attend to the target speaker in a complex acoustic environment, thus benefiting various downstream tasks. We further introduce a dual-path training strategy and use the cross-correlation constraint between the two branches to encourage the model to generate noise-invariant representation. Experiments on SUPERB benchmark and LibriMix dataset demonstrate the universality and noise-robustness of SHuBERT. Furthermore, we find that our high-quality representation can be easily integrated with conventional supervised learning methods to achieve significant performance, even under extremely low-resource labeled data.

20 August 2024 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
WWWupeng Wang
Wupeng Wang
ZPZexu Pan
Zexu Pan
XLXinke Li
Xinke Li
SWShuai Wang
Shuai Wang
HLHaizhou Li
Haizhou Li
Speech separation seeks to separate individual speech signals from a speech mixture. Typically, most separation models are trained on synthetic data due to the unavailability of target reference in real-world cocktail party scenarios. As a result, there exists a domain gap between real and synthetic data when deploying speech separation models in real-world applications. In this paper, we propose a self-supervised domain invariant pretrained (DIP) frontend that is exposed to mixture data without the need for target reference speech. The DIP frontend utilizes a Siamese network with two innovative pretext tasks, mixture predictive coding (MPC) and mixture invariant coding(MIC),to capture shared contextual cues between real and synthetic unlabeled mixtures. Subsequently, we freeze the DIP frontend as a feature extractor when training the downstream speech separation models on synthetic data. By pretraining the DIP frontend with the contextual cues, we expect that the speech separation skills learned from synthetic data can be effectively transferred to real data. To benefit from the DIP frontend, we introduce a novel separation pipeline to align the feature resolution of the separation models. We evaluate the speech separation quality on standard benchmarks and realworld datasets. The results confirm the superiority of our DIP frontend over existing speech separation models. This study underscores the potential of large-scale pretraining to enhance the quality and intelligibility of speech separation in real-world applications.

5 December 2025 in IEEE Transactions on Audio, Speech and Language Processing
Verified
Journal Article
Closed access
WWWupeng Wang
Wupeng Wang
ZPZexu Pan
Zexu Pan
XLXinke Li
Xinke Li
SWShuai Wang
Shuai Wang
HLHaizhou Li
Haizhou Li
Speech separation (SS) seeks to disentangle a multi-speaker speech mixture into single-speaker speech streams. Although SS can be generally achieved using offline methods, such a processing paradigm is not suitable for real-time streaming applications. Causal separation models, which rely only on past and present information,offer a promising solution for real-time streaming. However, these models typically suffer from notable performance degradation due to the absence of future context. In this paper, we introduce a novel frontend that is designed to mitigate the mismatch between training and run-time inference by implicitly incorporating future information into causal models during training through predictive patterns. The pretrained frontend employs a transformer decoder network with a causal convolutional encoder as the backbone and is pretrained in a self-supervised manner with two innovative pretext tasks: autoregressive hybrid prediction and contextual knowledge distillation. These tasks allow the model to capture predictive patterns directly from mixtures in a self-supervised manner. The pretrained frontend subsequently serves as a feature extractor to generate high-quality predictive patterns. Comprehensive evaluations on synthetic and real-world datasets are carried out to validate the effectiveness of the proposed pretrained frontend.

12 June 2026 in IEEE Transactions on Audio, Speech and Language Processing
Verified
Journal Article
Closed access
WWWupeng Wang
Wupeng Wang
ZPZexu Pan
Zexu Pan
JLJingru Lin
Jingru Lin
SWShuai Wang
Shuai Wang
HLHaizhou Li
Haizhou Li
Speech separation aims to isolate individual speech signals from multi-talker speech mixtures. Despite the impressive results on synthetic benchmarks, most systems suffer noticeable performance drop in face of real-world speech mixtures due to severe domain mismatch. To address this, we introduce a novel context-aware two-stage training scheme with iterative updates for speech separation models. In this training scheme, the conventional end-to-end architecture is replaced with a framework that contains a context extractor and a segregator. The two modules are trained iteratively to emulate the speech-separation process of the human auditory system. We evaluate the proposed training scheme through cross-domain experiments on both synthetic and real-world speech mixtures, and demonstrate that the proposed scheme effectively boosts separation quality across various domains without adaptation, as measured by signal quality metrics and word error rate (WER). Additionally, our ablation study on the real test set highlights that the perceptual objects, including phoneme and word representations from pretrained self-supervised learning (SSL) models, serve as effective training targets for separation models.