Shu-Tong Niu

Mr. Shu-Tong Niu

University of Science and Technology of China

Research Focus & Metrics

Academic Career Length

4Years (2023-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
20 August 2025 in IEEE Transactions on Audio, Speech and Language Processing
Verified
Journal Article
Closed access
SNShu-Tong Niu
Shu-Tong Niu
JDJun Du
Jun Du
RWRuo-Yu Wang
Ruo-Yu Wang
GYGao-Bin Yang
Gao-Bin Yang
TGTian Gao
Tian Gao
YHYu Hu
Yu Hu
We propose a single-channel Deep Cascade Fusion of Diarization and Separation (DCF-DS) framework11The open-source code will be released at https://github.com/rywang99/DCF-DS. for back-end automatic speech recognition (ASR), combining neural speaker diarization (NSD) and speech separation (SS). First, we sequentially integrate the NSD and SS modules within a joint training framework, enabling the separation module to leverage speaker time boundaries from the diarization module effectively. Then, to complement DCF-DS training, we introduce a window-level decoding scheme that allows the DCF-DS framework to handle the sparse data convergence instability (SDCI) problem. We also explore using an NSD system trained on real datasets to provide more accurate speaker boundaries. Additionally, we incorporate an optional multi-input multi-output speech enhancement module (MIMO-SE) within the DCF-DS framework, which offers further performance gains. Finally, we enhance diarization results by re-clustering DCF-DS outputs, improving ASR accuracy. By incorporating the DCF-DS method, we achieved first place in the realistic single-channel track of the CHiME-8 NOTSOFAR-1 challenge. We also perform the evaluation on the open LibriCSS dataset, achieving a new state-of-the-art single-channel speech recognition performance.

13 February 2025 in IEEE Transactions on Circuits and Systems for Video Technology
Verified
Journal Article
Closed access
KLKe-Wei Li
Ke-Wei Li
HCHang Chen
Hang Chen
JDJun Du
Jun Du
HZHeng-Shun Zhou
Heng-Shun Zhou
SSSabato Marco Siniscalchi
Sabato Marco Siniscalchi
SXShi-Fu Xiong
Shi-Fu Xiong
Audio-Visual Wake Word Spotting (AVWWS) aims to accurately detect user-defined keywords by leveraging the complementary nature of different modalities in challenging acoustic environments. However, two primary challenges hinder the application of AVWWS models in real-world scenarios: increased model parameters involving the video modality and the scarcity of paired audio-visual data. To address these issues, we propose a novel diverse acoustic knowledge distillation (DAKD) framework, which utilizes easily accessible single-modality audio data to train two teacher models and employs cross-modal knowledge distillation to transfer the generalization and de-noising capabilities of the teachers to the audio-visual student model. This approach mitigates the overfitting risk associated with large parameter counts and limited data. The DAKD framework consists of an audio-visual student model based on the lightweight multi-scale temporal-spatial attention (LMTSA) architecture, a multi-conditional teacher (MCT) model, and a de-noising teacher (DNT) model. The LMTSA model integrates compact 3D and 2D blocks based on the ResNet architecture through a simple attention module and accepts multi-scale supervision from word-level and phone-level labels, achieving joint temporal-spatial modeling with minimal parameter usage. The MCT and DNT models were trained using extensive real or simulated far-field speech and paired near-field and far-field speech, respectively, to generalize unseen acoustic environments and de-noising capabilities to the audio-visual student model. The effectiveness of our proposed DAKD framework is validated through comprehensive experiments on the MISP2021 and the updated MISP2021 Eval Hard datasets, establishing new benchmarks with fewer parameters. Our code will be available at https://github.com/wikkk-tp/AVWWS_DAKD.

13 February 2023 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
SNShu-Tong Niu
Shu-Tong Niu
JDJun Du
Jun Du
LSLei Sun
Lei Sun
YHYu Hu
Yu Hu
CLChin-Hui Lee
Chin-Hui Lee
We improve iterative separation-based speaker diarization (ISSD) with quality-aware dynamic masking (QDM). We call the proposed framework QDM-SSD. Compared with ISSD, QDM-SSD enhances the simulated data used for model adaptation through QDM to alleviate the influence of errors in speaker priors. In addition to data quality purification, QDM-SSD also makes the adaptation data sparse by automatically adjusting speaker overlap ratios according to data quality. Furthermore, using a sliding window over the adaptation data, clean regions in speech segments can be better localized. Experiments on the two-speaker conversational telephone speech (CTS) corpus show that the proposed QDM-SSD framework can reduce the diarization error rate (DER) by 18.56% relatively compared with ISSD. Moreover, QDM-SSD is shown to generalize to other two-speaker non-conversation telephone speech data sets where ISSD fails to work. Finally, we demonstrate that QDM-SSD can serve as a front-end to improve the performances of back-end automatic speech recognition.

23 March 2023 in Applied Sciences
Verified
Journal Article
Open access
PDF file
BYBing Yin
Bing Yin
SNShu-Tong Niu
Shu-Tong Niu
HTHaitao Tang
Haitao Tang
LSLei Sun
Lei Sun
JDJun Du
Jun Du
CLCong Liu
Cong Liu
Robust speech recognition in real world situations is still an important problem, especially when it is affected by environmental interference factors and conversational multi-speaker interactions. Supplementing audio information with other modalities, such as audio–visual speech recognition (AVSR), is a promising direction for improving speech recognition. The end-to-end (E2E) framework can learn information between multiple modalities well; however, the model is not easy to train, especially when the amount of data is relatively small. In this paper, we focus on building an encoder–decoder-based end-to-end audio–visual speech recognition system for use under realistic scenarios. First, we discuss different pre-training methods which provide various kinds of initialization for the AVSR framework. Second, we explore different model architectures and audio–visual fusion methods. Finally, we evaluate the performance on the corpus from the first Multi-modal Information based Speech Processing (MISP) challenge, which is recorded in a real home television (TV) room. By system fusion, our final system achieves a 23.98% character error rate (CER), which is better than the champion system of the first MISP challenge (CER = 25.07%).

27 June 2023 in Speech Communication
Verified
Journal Article
Closed access
SCShi Cheng
Shi Cheng
JDJun Du
Jun Du
SNShu-Tong Niu
Shu-Tong Niu
ACAlejandrina Cristia
Alejandrina Cristia
XWXin Wang
Xin Wang
CLChin-Hui Lee
Chin-Hui Lee