In this paper, we describe the ATR multilingual speech-to-speech translation (S2ST) system, which is mainly focused on translation between English and Asian languages (Japanese and Chinese). There are three main modules of our S2ST system: large-vocabulary continuous speech recognition, machine text-to-text (T2T) translation, and text-to-speech synthesis. All of them are multilingual and are designed using state-of-the-art technologies developed at ATR. A corpus-based statistical machine learning framework forms the basis of our system design. We use a parallel multilingual database consisting of over 600 000 sentences that cover a broad range of travel-related conversations. Recent evaluation of the overall system showed that speech-to-speech translation quality is high, being at the level of a person having a Test of English for International Communication (TOEIC) score of 750 out of the perfect score of 990.

As an indispensable module of computer-aided pronunciation training (CAPT) systems, mispronunciation detection and diagnosis (MDD) techniques have attracted a lot of attention from academia and industry over the past decade. To train robust MDD models, this technique requires massive human-annotated speech recordings which are usually expensive and even hard to acquire. In this study, we propose to use transfer learning to tackle the problem of data scarcity from two aspects. First, from audio modality, we explore the use of the pretrained model wav2vec2.0 for MDD tasks by learning robust general acoustic representation. Second, from text modality, we explore transferring prior texts into MDD by learning associations between acoustic and textual modalities. We propose textual modulation gates that assign more importance to the relevant text information while suppressing irrelevant text information. Moreover, given the transcriptions, we propose an extra contrastive loss to reduce the difference of learning objectives between the phoneme recognition and MDD tasks. Conducting experiments on the L2-Arctic dataset showed that our wav2vec2.0 based models outperformed the conventional methods. The proposed textual modulation gate and contrastive loss further improved the F1-score by more than 2.88% and our best model achieved an F1-score of 61.75%.

Phonetic contrasts (PC) are the basis of human speech communication. Different PCs have different levels of importance, which can be evaluated by the measurement of Functional Load (FL). It has also been understood that human speech perception is dependent not only on PCs but also on high-level context such as word, word sequence, grammar and etc. Thus it is desirable to take into account those context effects when estimating FLs. For this purpose, we propose in this paper a new method to meet the requirement, which estimates the FL based on Mutual Information (MI) of text and phonemes. It is capable of estimating FLs of any kinds of PCs in the context of a word sequence. Preliminary experiments were done using Chinese newspaper text corpus. The results showed to be very meaningful.

We present in this paper a new Chinese lexical tone recognition approach based on our pitch anchoring hypothesis, which suggests that the tone offset of the preceding lexical tone and the tone onset of the succeeding lexical tone serve as anchor points for the pitch heights of the onset and offset of the sandwiched lexical tone. The new approach exploits relative F0 heights between neighboring tones as important discriminating features for the lexical tones. Experimental results revealed that the new approach could increase the tone recognition accuracy greatly: above 10% (from 75.3% to 85.5%) compared with the conventional one.