19 September 2023 in IEEE/ACM Transactions on Audio, Speech, and Language Processing
Directly training a document-to-document (Doc2Doc) neural machine translation (NMT) via Transformer from scratch, especially on small datasets, usually fails to converge. Our dedicated probing tasks show that 1) both the absolute position and relative position information gets gradually weakened or even vanished once it reaches the upper encoder layers, and 2) the vanishing of absolute position information in encoder output causes the training failure of Doc2Doc NMT. To alleviate this problem, we propose a position-aware Transformer (P-Transformer) to enhance both the absolute and relative position information in both self-attention and cross-attention. Specifically, we integrate absolute positional information, i.e., position embeddings, into the query-key pairs both in self-attention and cross-attention through a simple yet effective addition operation. Moreover, we also integrate relative position encoding in self-attention. The proposed P-Transformer utilizes sinusoidal position encoding and does not require any task-specified position embedding, segment embedding, or attention mechanism. Through the above methods, we build a Doc2Doc NMT model with P-Transformer, which ingests the source document and completely generates the target document in a sequence-to-sequence (seq2seq) way. In addition, P-Transformer can be applied to seq2seq-based document-to-sentence (Doc2Sent) and sentence-to-sentence (Sent2Sent) translations. Extensive experimental results of Doc2Doc NMT show that P-Transformer significantly outperforms strong baselines on the widely-used 9 document-level datasets in 7 language pairs, covering small-, middle-, and large-scales, and achieves a new state-of-the-art. Experimentation on discourse phenomena shows that our Doc2Doc NMT models improve the translation quality in both BLEU and discourse coherence. We make our code available on Github.
5 January 2024 in Journal of Molecular Graphics and Modelling
13 November 2025 in Neurocomputing
1 November 2020 in Journal of Physics: Conference Series
In recent years, the proposal of neural network has provided new idea for solving natural language processing, and at the same time, neural machine translation has become the frontier method of machine translation. In low-resource languages, due to the sparse bilingual data, the model needs more high-quality data, and the translation quality fails to achieve the desired effect. In this paper, experiments on neural network machine translation based on attention are conducted on Tibetan-Chinese language pairs, and transfer learning method combined with back translation method is used to alleviate the problem of insufficient Tibetan-Chinese parallel corpus. Experimental results show that the proposed transfer learning combined with back translation method is simple and effective. Compared with traditional translation methods, the translation effect is significantly improved. From the analysis of translation, it can be seen that the citation of Tibetan-Chinese neural machine translation is smoother, which is greatly improved compared to the translation without back translation. At the same time, there are common deficiencies in neural machine translation such as inadequate translation and low translation loyalty.