Shinji Watanabe

Shinji Watanabe

Top Cited Scholar 2026
Auto-generated profile, based on publicly available information

Shinji Watanabe

Carnegie Mellon University, Pittsburgh, PA, USA

Research Focus & Metrics

Academic Career Length

28Years (1999-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
2 September 2018
Verified
Conference Paper
Closed access
SWShinji Watanabe
Shinji Watanabe
THTakaaki Hori
Takaaki Hori
SKShigeki Karita
Shigeki Karita
THTomoki Hayashi
Tomoki Hayashi
JNJiro Nishitoba
Jiro Nishitoba
TOTsubasa Ochiai
Tsubasa Ochiai

16 October 2017 in IEEE Journal of Selected Topics in Signal Processing
Verified
Journal Article
Closed access
SWShinji Watanabe
Shinji Watanabe
THTakaaki Hori
Takaaki Hori
SKSuyoun Kim
Suyoun Kim
JHJohn R. Hershey
John R. Hershey
THTomoki Hayashi
Tomoki Hayashi
Conventional automatic speech recognition (ASR) based on a hidden Markov model (HMM)/deep neural network (DNN) is a very complicated system consisting of various modules such as acoustic, lexicon, and language models. It also requires linguistic resources, such as a pronunciation dictionary, tokenization, and phonetic context-dependency trees. On the other hand, end-to-end ASR has become a popular alternative to greatly simplify the model-building process of conventional ASR systems by representing complicated modules with a single deep network architecture, and by replacing the use of linguistic resources with a data-driven learning method. There are two major types of end-to-end architectures for ASR; attention-based methods use an attention mechanism to perform alignment between acoustic frames and recognized symbols, and connectionist temporal classification (CTC) uses Markov assumptions to efficiently solve sequential problems by dynamic programming. This paper proposes hybrid CTC/attention end-to-end ASR, which effectively utilizes the advantages of both architectures in training and decoding. During training, we employ the multiobjective learning framework to improve robustness and achieve fast convergence. During decoding, we perform joint decoding by combining both attention-based and CTC scores in a one-pass beam search algorithm to further eliminate irregular alignments. Experiments with English (WSJ and CHiME-4) tasks demonstrate the effectiveness of the proposed multiobjective learning over both the CTC and attention-based encoder-decoder baselines. Moreover, the proposed method is applied to two large-scale ASR benchmarks (spontaneous Japanese and Mandarin Chinese), and exhibits performance that is comparable to conventional DNN/HMM ASR systems based on the advantages of both multiobjective learning and joint decoding without linguistic resources.

1 March 2016
Verified
Conference Paper
Closed access
JHJohn R. Hershey
John R. Hershey
ZCZhuo Chen
Zhuo Chen
JRJonathan Le Roux
Jonathan Le Roux
SWShinji Watanabe
Shinji Watanabe

1 March 2017
Verified
Conference Paper
Closed access
SKSuyoun Kim
Suyoun Kim
THTakaaki Hori
Takaaki Hori
SWShinji Watanabe
Shinji Watanabe
Recently, there has been an increasing interest in end-to-end speech recognition that directly transcribes speech to text without any predefined alignments. One approach is the attention-based encoder-decoder framework that learns a mapping between variable-length input and output sequences in one step using a purely data-driven method. The attention model has often been shown to improve the performance over another end-to-end approach, the Connectionist Temporal Classification (CTC), mainly because it explicitly uses the history of the target character without any conditional independence assumptions. However, we observed that the performance of the attention has shown poor results in noisy condition and is hard to learn in the initial training stage with long input sequences. This is because the attention model is too flexible to predict proper alignments in such cases due to the lack of left-to-right constraints as used in CTC. This paper presents a novel method for end-to-end speech recognition to improve robustness and achieve fast convergence by using a joint CTC-attention model within the multi-task learning framework, thereby mitigating the alignment issue. An experiment on the WSJ and CHiME-4 tasks demonstrates its advantages over both the CTC and attention-based encoder-decoder baselines, showing 5.4-14.6% relative improvements in Character Error Rate (CER).

15 September 2022 in IEEE Journal of Selected Topics in Signal Processing
Verified
Journal Article
Closed access
AMAbdelrahman Mohamed
Abdelrahman Mohamed
HLHung-Yi Lee
Hung-Yi Lee
LBLasse Borgholt
Lasse Borgholt
JHJakob D. Havtorn
Jakob D. Havtorn
JEJoakim Edin
Joakim Edin
SWShinji Watanabe
Shinji Watanabe
Although supervised deep learning has revolutionized speech and audio processing, it has necessitated the building of specialist models for individual tasks and application scenarios. It is likewise difficult to apply this to dialects and languages for which only limited labeled data is available. Self-supervised representation learning methods promise a single universal model that would benefit a wide variety of tasks and domains. Such methods have shown success in natural language processing and computer vision domains, achieving new levels of performance while reducing the number of labels required for many downstream scenarios. Speech representation learning is experiencing similar progress in three main categories: generative, contrastive, and predictive methods. Other approaches rely on multi-modal data for pre-training, mixing text or visual data streams with speech. Although self-supervised speech representation is still a nascent research area, it is closely related to acoustic word embedding and learning with zero lexical resources, both of which have seen active research for many years. This review presents approaches for self-supervised speech representation learning and their connection to other research areas. Since many current methods focus solely on automatic speech recognition as a downstream task, we review recent efforts on benchmarking learned representations to extend the application beyond speech recognition.

1 April 2015
Verified
Conference Paper
Closed access
HEHakan Erdogan
Hakan Erdogan
JHJohn R. Hershey
John R. Hershey
SWShinji Watanabe
Shinji Watanabe
JRJonathan Le Roux
Jonathan Le Roux
Separation of speech embedded in non-stationary interference is a challenging problem that has recently seen dramatic improvements using deep network-based methods. Previous work has shown that estimating a masking function to be applied to the noisy spectrum is a viable approach that can be improved by using a signal-approximation based objective function. Better modeling of dynamics through deep recurrent networks has also been shown to improve performance. Here we pursue both of these directions. We develop a phase-sensitive objective function based on the signal-to-noise ratio (SNR) of the reconstructed signal, and show that in experiments it yields uniformly better results in terms of signal-to-distortion ratio (SDR). We also investigate improvements to the modeling of dynamics, using bidirectional recurrent networks, as well as by incorporating speech recognition outputs in the form of alignment vectors concatenated with the spectral input features. Both methods yield further improvements, pointing to tighter integration of recognition with separation as a promising future direction.