Hang Lv

Hang Lv

Auto-generated profile, based on publicly available information

Hang Lv

ASLP@NPU, School of Computer Science and Engineering Northwestern Polytechnical University Xi'an PR China

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
9 December 2024 in IEEE Transactions on Audio, Speech and Language Processing
Verified
Journal Article
Closed access
PGPengcheng Guo
Pengcheng Guo
XCXuankai Chang
Xuankai Chang
HLHang Lv
Hang Lv
SWShinji Watanabe
Shinji Watanabe
LXLei Xie
Lei Xie
Benefiting from massive and diverse data sources, speech foundation models exhibit strong generalization and knowledge transfer capabilities to a wide range of downstream tasks. However, a limitation arises from their exclusive handling of single-speaker speech input, making them ineffective in recognizing multi-speaker overlapped speech, a common occurrence in real-world scenarios. In this study, we delve into the adaptation of speech foundation models to eliminate interfering speakers from overlapping speech and perform target-speaker automatic speech recognition (TS-ASR). Initially, we utilize the Whisper model as the foundation for adaptation and conduct a thorough comparison of its integration with existing target-speaker adaptation techniques. We then propose an innovative model termed Speaker-Querying Whisper (SQ-Whisper), which employs a set number of trainable queries to capture speaker prompts from overlapping speech based on target-speaker enrollment. These prompts serve to steer the model in extracting speaker-specific features and accurately recognizing target-speaker transcriptions. Experimental results demonstrate that our approach effectively adapts the pre-trained speech foundation model to TS-ASR. Compared with the robust TS-HuBERT model, the proposed SQ-Whisper significantly improves performance, yielding up to 15% and 10% relative reductions in word error rates (WERs) on the Libri2Mix and WSJ0-2Mix datasets, respectively. With data augmentation, we establish new state-of-the-art WERs of 14.6% on the Libri2Mix Test set and 4.4% on the WSJ0-2Mix Test set. Furthermore, we evaluate our model on the real-world AMI meeting dataset, which shows consistent improvement over other adaptation methods.

16 April 2024 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
KWKun Wei
Kun Wei
BLBei Li
Bei Li
HLHang Lv
Hang Lv
QLQuan Lu
Quan Lu
NJNing Jiang
Ning Jiang
LXLei Xie
Lei Xie
Automatic Speech Recognition (ASR) in conversational settings presents unique challenges, including extracting relevant contextual information from previous conversational turns. Due to irrelevant content, error propagation, and redundancy, existing methods struggle to extract longer and more effective contexts. To address this issue, we introduce a novel conversational ASR system, extending the Conformer encoder-decoder model with cross-modal conversational representation. Our approach leverages a cross-modal extractor that combines pre-trained speech and text models through a specialized encoder and a modal-level mask input. This enables the extraction of richer historical speech context without explicit error propagation. We also incorporate conditional latent variational modules to learn conversational-level attributes such as role preference and topic coherence. By introducing both cross-modal and conversational representations into the decoder, our model retains longer context without information loss, achieving relative accuracy improvements of 8.8% and 23% on Mandarin conversation datasets HKUST and MagicData-RAMC, respectively, compared to the standard Conformer model.

23 February 2022 in arXiv
Verified
Preprint
Open access
PDF file
BZBinbin Zhang
Binbin Zhang
HLHang Lv
Hang Lv
PGPengcheng Guo
Pengcheng Guo
QSQijie Shao
Qijie Shao
CYChao Yang
Chao Yang
ZPZhendong Peng
Zhendong Peng
In this paper, we present WenetSpeech, a multi-domain Mandarin corpus consisting of 10000+ hours high-quality labeled speech, 2400+ hours weakly labeled speech, and about 10000 hours unlabeled speech, with 22400+ hours in total. We collect the data from YouTube and Podcast, which covers a variety of speaking styles, scenarios, domains, topics, and noisy conditions. An optical character recognition (OCR) based method is introduced to generate the audio/text segmentation candidates for the YouTube data on its corresponding video captions, while a high-quality ASR transcription system is used to generate audio/text pair candidates for the Podcast data. Then we propose a novel end-to-end label error detection approach to further validate and filter the candidates. We also provide three manually labelled high-quality test sets along with WenetSpeech for evaluation -- Dev for cross-validation purpose in training, Test_Net, collected from Internet for matched test, and Test\_Meeting, recorded from real meetings for more challenging mismatched test. Baseline systems trained with WenetSpeech are provided for three popular speech recognition toolkits, namely Kaldi, ESPnet, and WeNet, and recognition results on the three test sets are also provided as benchmarks. To the best of our knowledge, WenetSpeech is the current largest open-sourced Mandarin speech corpus with transcriptions, which benefits research on production-level speech recognition.

18 March 2021 in IEEE Signal Processing Letters
Verified
Journal Article
Closed access
HLHang Lv
Hang Lv
DPDaniel Povey
Daniel Povey
MYMahsa Yarmohammadi
Mahsa Yarmohammadi
KLKe Li
Ke Li
YWYiming Wang
Yiming Wang
SKSanjeev Khudanpur
Sanjeev Khudanpur
We propose a novel lazy-evaluation token-group decoding algorithm with on-the-fly composition of weighted finite-state transducers (WFSTs) for large vocabulary continuous speech recognition. In the standard on-the-fly composition decoder, a base WFST and one or more incremental WFSTs are composed during decoding, and then token passing algorithm is employed to generate the lattice on the composed search space, resulting in substantial computation overhead. To improve speed, the proposed algorithm adopts 1) a token-group method, which groups tokens with the same state in the base WFST on each frame and limits the capacity of the group and 2) a lazy-evaluation method, which does not expand a token group and its source token groups until it processes a word label during decoding.Experiments show that the proposed decoder works notably up to 3 times faster than the standard on-the-fly composition decoder.

5 July 2022 in arXiv
Verified
Preprint
Open access
PDF file
BZBinbin Zhang
Binbin Zhang
DWDi Wu
Di Wu
ZPZhendong Peng
Zhendong Peng
XSXingchen Song
Xingchen Song
ZYZhuoyuan Yao
Zhuoyuan Yao
JNJianwei Niu
Jianwei Niu
Recently, we made available WeNet, a production-oriented end-to-end speech recognition toolkit, which introduces a unified two-pass (U2) framework and a built-in runtime to address the streaming and non-streaming decoding modes in a single model. To further improve ASR performance and facilitate various production requirements, in this paper, we present WeNet 2.0 with four important updates. (1) We propose U2++, a unified two-pass framework with bidirectional attention decoders, which includes the future contextual information by a right-to-left attention decoder to improve the representative ability of the shared encoder and the performance during the rescoring stage. (2) We introduce an n-gram based language model and a WFST-based decoder into WeNet 2.0, promoting the use of rich text data in production scenarios. (3) We design a unified contextual biasing framework, which leverages user-specific context (e.g., contact lists) to provide rapid adaptation ability for production and improves ASR accuracy in both with-LM and without-LM scenarios. (4) We design a unified IO to support large-scale data for effective model training. In summary, the brand-new WeNet 2.0 achieves up to 10\% relative recognition performance improvement over the original WeNet on various corpora and makes available several important production-oriented features.

1 January 2025 in Electronics Letters
Verified
Journal Article
Open access
FLFangyi Li
Fangyi Li
HLHang Lv
Hang Lv
YWYiming Wang
Yiming Wang
LXLei Xie
Lei Xie
Neural language models (NLMs) have been shown to outperform n‐gram language models in automatic speech recognition (ASR) tasks. NLMs are usually used in the second‐pass lattice rescoring rather than the first‐pass decoding, since its encoded infinite history virtually cannot be compiled into static decoding graphs. However, the modeling power of NLMs is not fully leveraged due to the constraints imposed by the lattice, leading to accuracy loss. To improve this, on‐the‐fly composition decoders were proposed to utilize NLMs in first‐pass decoding with increased computational cost. In this paper, an asynchronous lazy‐evaluation token‐group decoder with exact lattice generation is proposed to reduce the computational cost of the on‐the‐fly composition decoder, achieving significant decoding speedup. More specifically, having a novel token‐group with a representative element data structure, the proposed decoder performs lazy‐evaluation which expands the tokens until a word boundary is reached. Furthermore, based on the score of the representative element in a token‐group, the decoder prunes unpromising tokens by an A* algorithm. The experiments show that the proposed decoder can accelerate the vanilla on‐the‐fly composition decoder by up to 6.9 times, and get paths with even better average likelihoods than lattice rescoring approaches.