Shang-Wen Li

Shang-Wen Li

Auto-generated profile, based on publicly available information

Shang-Wen Li

Amazon AI

Research Focus & Metrics

Academic Career Length

7Years (2020-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
8 July 2021 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
ALAndy T. Liu
Andy T. Liu
SLShang-Wen Li
Shang-Wen Li
HLHung-Yi Lee
Hung-Yi Lee
We introduce a self-supervised speech pre-training method called TERA, which stands for Transformer Encoder Representations from Alteration. Recent approaches often learn by using a single auxiliary task like contrastive prediction, autoregressive prediction, or masked reconstruction. Unlike previous methods, we use alteration along three orthogonal axes to pre-train Transformer Encoders on a large amount of unlabeled speech. The model learns through the reconstruction of acoustic frames from their altered counterpart, where we use a stochastic policy to alter along various dimensions: time, frequency, and magnitude. TERA can be used for speech representations extraction or fine-tuning with downstream models. We evaluate TERA on several downstream tasks, including phoneme classification, keyword spotting, speaker recognition, and speech recognition. We present a large-scale comparison of various self-supervised models. TERA achieves strong performance in the comparison by improving upon surface features and outperforming previous models. In our experiments, we study the effect of applying different alteration techniques, pre-training on more data, and pre-training on various features. We analyze different model sizes and find that smaller models are strong representation learners than larger models, while larger models are more effective for downstream fine-tuning than smaller models. Furthermore, we show the proposed method is transferable to downstream datasets not used in pre-training.

16 April 2024 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
SYShu-Wen Yang
Shu-Wen Yang
HCHeng-Jui Chang
Heng-Jui Chang
ZHZili Huang
Zili Huang
ALAndy T. Liu
Andy T. Liu
CLCheng-I Lai
Cheng-I Lai
HLHung-Yi Lee
Hung-Yi Lee
The foundation model paradigm leverages a shared foundation model to achieve state-of-the-art (SOTA) performance for various tasks, requiring minimal downstream-specific data collection and modeling. This approach has proven crucial in the field of Natural Language Processing (NLP). However, the speech processing community lacks a similar setup to explore the paradigm systematically. To bridge this gap, we establish the Speech processing Universal PERformance Benchmark (SUPERB). SUPERB represents an ecosystem designed to evaluate foundation models across a wide range of speech processing tasks, facilitating the sharing of results on an online leaderboard and fostering collaboration through a community-driven benchmark database that aids in new development cycles. We present a unified learning framework for solving the speech processing tasks in SUPERB with the frozen foundation model followed by task-specialized lightweight prediction heads. Combining our results with community submissions, we verify that the framework is simple yet effective, as the best-performing foundation model shows competitive generalizability across most SUPERB tasks. Finally, we conduct a series of analyses to offer an in-depth understanding of SUPERB and speech foundation models, including information flows across tasks inside the models and the statistical significance and robustness of the benchmark.

1 January 2022
Verified
Conference Paper
Open access
PDF file
HLHung-Yi Lee
Hung-Yi Lee
SLShang-Wen Li
Shang-Wen Li
TVThang Vu
Thang Vu
Hung-yi Lee, Shang-Wen Li, Thang Vu. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

1 January 2022
Verified
Conference Paper
Open access
PDF file
XJXisen Jin
Xisen Jin
DZDejiao Zhang
Dejiao Zhang
HZHenghui Zhu
Henghui Zhu
WXWei Xiao
Wei Xiao
SLShang-Wen Li
Shang-Wen Li
XRXiang Ren
Xiang Ren
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, Xiang Ren. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

9 January 2023
Verified
Conference Paper
Open access
TFTzu-Hsun Feng
Tzu-Hsun Feng
ADAnnie Dong
Annie Dong
CYChing-Feng Yeh
Ching-Feng Yeh
SYShu-Wen Yang
Shu-Wen Yang
TLTzu-Quan Lin
Tzu-Quan Lin
HLHung-Yi Lee
Hung-Yi Lee
We present the SUPERB challenge at SLT 2022, which aims at learning self-supervised speech representation for better performance, generalization, and efficiency. The challenge builds upon the SUPERB benchmark and implements metrics to measure the computation requirements of self-supervised learning (SSL) representation and to evaluate its generalizability and performance across the diverse SUPERB tasks. The SUPERB benchmark provides comprehensive coverage of popular speech processing tasks, from speech and speaker recognition to audio generation and semantic understanding. As SSL has gained interest in the speech community and showed promising outcomes, we envision the challenge to uplevel the impact of SSL techniques by motivating more practical designs of techniques beyond task performance. We summarize the results of 14 submitted models in this paper. We also discuss the main findings from those submissions and the future directions of SSL research.

2 August 2024 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
KCKai-Wei Chang
Kai-Wei Chang
HWHaibin Wu
Haibin Wu
YWYu-Kai Wang
Yu-Kai Wang
YWYuan-Kuei Wu
Yuan-Kuei Wu
HSHua Shen
Hua Shen
HLHung-Yi Lee
Hung-Yi Lee
Prompting has become a practical method for utilizing pre-trained language models (LMs). This approach offers several advantages. It allows an LM to adapt to new tasks with minimal training and parameter updates, thus achieving efficiency in both storage and computation. Additionally, prompting modifies only the LM's inputs and harnesses the generative capabilities of language models to address various downstream tasks in a unified manner. This significantly reduces the need for human labor in designing task-specific models. These advantages become even more evident as the number of tasks served by the LM scales up. Motivated by the strengths of prompting, we are the first to explore the potential of prompting speech LMs in the domain of speech processing. Recently, there has been a growing interest in converting speech into discrete units for language modeling. Our pioneer research demonstrates that these quantized speech units are highly versatile within our unified prompting framework. Not only can they serve as class labels, but they also contain rich phonetic information that can be re-synthesized back into speech signals for speech generation tasks. Specifically, we reformulate speech processing tasks into speech-to-unit generation tasks. As a result, we can seamlessly integrate tasks such as speech classification, sequence generation, and speech generation within a single, unified prompting framework. The experiment results show that the prompting method can achieve competitive performance compared to the strong fine-tuning method based on self-supervised learning models with a similar number of trainable parameters. The prompting method also shows promising results in the few-shot setting. Moreover, with the advanced speech LMs coming into the stage, the proposed prompting framework attains great potential.