9 August 2018 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Recently, researchers have shown an increasing interest in incorporating linguistic knowledge into neural machine translation (NMT). To this end, previous works choose either to alter the architecture of NMT encoder to incorporate syntactic information into the translation model, or to generalize the embedding layer of the encoder to encode additional linguistic features. The former approach mainly focuses on injecting the syntactic structure of the source sentence into the encoding process, leading to a complicated model that lacks the flexibility to incorporate other types of knowledge. The latter extends word embeddings by considering additional linguistic knowledge as features to enrich the word representation. It thus does not explicitly balance the contribution from word embeddings and the contribution from additional linguistic knowledge. To address these limitations, this paper proposes a knowledge-aware NMT approach that models additional linguistic features in parallel to the word feature. The core idea is that we propose modeling a series of linguistic features at the word level (knowledge block) using a recurrent neural network (RNN). And in sentence level, those word-corresponding feature blocks are further encoded using a RNN encoder. In decoding, we propose a knowledge gate and an attention gate to dynamically control the proportions of information contributing to the generation of target words from different sources. Extensive experiments show that our approach is capable of better accounting for importance of additional linguistic, and we observe significant improvements from 1.0 to 2.3 BLEU points on Chinese $\leftrightarrow$ English and English $\rightarrow$ German translation tasks.
17 May 2023 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
In neural machine translation (NMT), most sequence-to-sequence (seq2seq) models are trained only with the teacher-forcing paradigm, where the ground truth history is used to predict the next ground truth word. At the inference stage, however, the decoder predicts the next token solely based on history generated from scratch. Both using ground truth history and predicting ground truth words potentially lead to exposure bias. On the one hand, to alleviate the issue of exposure bias caused by using ground truth history, we propose contextual augmentation by allowing substitution, insertion, and deletion of words. The contextual augmentation applies to target sequence to generate non-ground truth and natural history when predicting next words. On the other hand, to alleviate the exposure bias caused by predicting ground truth words, we further apply self distillation to guide the model to carry out optimization according to smoothed prediction distribution, i.e, enable the model to predict not only ground truth words, but also other potentially correct and reasonable words. Experimental results on WMT14 English $\leftrightarrow$ German and IWSLT14 German $\rightarrow$ English translation tasks demonstrate that our approach achieves significant improvements over Transformer on standard benchmarks. Detailed experimental analyses further reveal the effectiveness of our proposed approach on improving the translation quality.