Hung-Yi Lee

Hung-Yi Lee

Auto-generated profile, based on publicly available information

Hung-Yi Lee

National Taiwan University, Taipei, Taiwan

Research Focus & Metrics

Academic Career Length

18Years (2009-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
15 September 2022 in IEEE Journal of Selected Topics in Signal Processing
Verified
Journal Article
Closed access
AMAbdelrahman Mohamed
Abdelrahman Mohamed
HLHung-Yi Lee
Hung-Yi Lee
LBLasse Borgholt
Lasse Borgholt
JHJakob D. Havtorn
Jakob D. Havtorn
JEJoakim Edin
Joakim Edin
SWShinji Watanabe
Shinji Watanabe
Although supervised deep learning has revolutionized speech and audio processing, it has necessitated the building of specialist models for individual tasks and application scenarios. It is likewise difficult to apply this to dialects and languages for which only limited labeled data is available. Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Such methods have shown success in natural language processing and computer vision domains, achieving new levels of performance while reducing the number of labels required for many downstream scenarios. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. Other approaches rely on multi-modal data for pre-training, mixing text or visual data streams with speech. Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years. This review presents approaches for self-supervised speech representation learning and their connection to other research areas. Since many current methods focus solely on automatic speech recognition as a downstream task, we review recent efforts on benchmarking learned representations to extend the application beyond speech recognition.

8 July 2021 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
ALAndy T. Liu
Andy T. Liu
SLShang-Wen Li
Shang-Wen Li
HLHung-Yi Lee
Hung-Yi Lee
We introduce a self-supervised speech pre-training method called TERA, which stands for Transformer Encoder Representations from Alteration. Recent approaches often learn by using a single auxiliary task like contrastive prediction, autoregressive prediction, or masked reconstruction. Unlike previous methods, we use alteration along three orthogonal axes to pre-train Transformer Encoders on a large amount of unlabeled speech. The model learns through the reconstruction of acoustic frames from their altered counterpart, where we use a stochastic policy to alter along various dimensions: time, frequency, and magnitude. TERA can be used for speech representations extraction or fine-tuning with downstream models. We evaluate TERA on several downstream tasks, including phoneme classification, keyword spotting, speaker recognition, and speech recognition. We present a large-scale comparison of various self-supervised models. TERA achieves strong performance in the comparison by improving upon surface features and outperforming previous models. In our experiments, we study the effect of applying different alteration techniques, pre-training on more data, and pre-training on various features. We analyze different model sizes and find that smaller models are strong representation learners than larger models, while larger models are more effective for downstream fine-tuning than smaller models. Furthermore, we show the proposed method is transferable to downstream datasets not used in pre-training.

1 September 2015 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Verified
Journal Article
Closed access
LLLin-Shan Lee
Lin-Shan Lee
JGJames Glass
James Glass
HLHung-Yi Lee
Hung-Yi Lee
CCChun-An Chan
Chun-An Chan
Spoken content retrieval refers to directly indexing and retrieving spoken content based on the audio rather than text descriptions. This potentially eliminates the requirement of producing text descriptions for multimedia content for indexing and retrieval purposes, and is able to precisely locate the exact time the desired information appears in the multimedia. Spoken content retrieval has been very successfully achieved with the basic approach of cascading automatic speech recognition (ASR) with text information retrieval: after the spoken content is transcribed into text or lattice format, a text retrieval engine searches over the ASR output to find desired information. This framework works well when the ASR accuracy is relatively high, but becomes less adequate when more challenging real-world scenarios are considered, since retrieval performance depends heavily on ASR accuracy. This challenge leads to the emergence of another approach to spoken content retrieval: to go beyond the basic framework of cascading ASR with text retrieval in order to have retrieval performances that are less dependent on ASR accuracy. This overview article is intended to provide a thorough overview of the concepts, principles, approaches, and achievements of major technical contributions along this line of investigation. This includes five major directions: 1) Modified ASR for Retrieval Purposes: cascading ASR with text retrieval, but the ASR is modified or optimized for spoken content retrieval purposes; 2) Exploiting the Information not present in ASR outputs: to try to utilize the information in speech signals inevitably lost when transcribed into phonemes and words; 3) Directly Matching at the Acoustic Level without ASR: for spoken queries, the signals can be directly matched at the acoustic level, rather than at the phoneme or word levels, bypassing all ASR issues; 4) Semantic Retrieval of Spoken Content: trying to retrieve spoken content that is semantically related to the query, but not necessarily including the query terms themselves; 5) Interactive Retrieval and Efficient Presentation of the Retrieved Objects: with efficient presentation of the retrieved objects, an interactive retrieval process incorporating user actions may produce better retrieval results and user experiences.

1 January 2023
Verified
Conference Paper
Open access
PDF file
CCCheng-Han Chiang
Cheng-Han Chiang
HLHung-Yi Lee
Hung-Yi Lee
Cheng-Han Chiang, Hung-yi Lee. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.

1 January 2019
Verified
Conference Paper
Open access
PDF file
YWYaushian Wang
Yaushian Wang
HLHung-Yi Lee
Hung-Yi Lee
Yun-Nung (Vivian) ChenYun-Nung (Vivian) Chen
Yun-Nung (Vivian) Chen
Yaushian Wang, Hung-Yi Lee, Yun-Nung Chen. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

1 January 2024
Verified
Conference Paper
Open access
PDF file
ZTZhi Rui Tam
Zhi Rui Tam
CWCheng-Kuang Wu
Cheng-Kuang Wu
YTYi-Lin Tsai
Yi-Lin Tsai
CLChieh-Yen Lin
Chieh-Yen Lin
HLHung-Yi Lee
Hung-Yi Lee
Yun-Nung (Vivian) ChenYun-Nung (Vivian) Chen
Yun-Nung (Vivian) Chen
Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, Yun-Nung Chen. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track. 2024.