Chenze Shao

Chenze Shao

Auto-generated profile, based on publicly available information

Chenze Shao

Institute of Computing Technology Key Laboratory of Intelligent Information Processing Chinese Academy of Sciences

Research Focus & Metrics

Academic Career Length

8Years (2019-Present)

(Calculated once Scilit publication data is synced: current year − year of first publication + 1)

Representative Publications

View All Publications
23 December 2021 in Computational Linguistics
Verified
Journal Article
Open access
PDF file
CSChenze Shao
Chenze Shao
Yan FengYan Feng
Yan Feng
JZJinchao Zhang
Jinchao Zhang
FMFandong Meng
Fandong Meng
JZJie Zhou
Jie Zhou
In recent years, Neural Machine Translation (NMT) has achieved notable results in various translation tasks. However, the word-by-word generation manner determined by the autoregressive mechanism leads to high translation latency of the NMT and restricts its low-latency applications. Non-Autoregressive Neural Machine Translation (NAT) removes the autoregressive mechanism and achieves significant decoding speedup by generating target words independently and simultaneously. Nevertheless, NAT still takes the word-level cross-entropy loss as the training objective, which is not optimal because the output of NAT cannot be properly evaluated due to the multimodality problem. In this article, we propose using sequence-level training objectives to train NAT models, which evaluate the NAT outputs as a whole and correlates well with the real translation quality. First, we propose training NAT models to optimize sequence-level evaluation metrics (e.g., BLEU) based on several novel reinforcement algorithms customized for NAT, which outperform the conventional method by reducing the variance of gradient estimation. Second, we introduce a novel training objective for NAT models, which aims to minimize the Bag-of-N-grams (BoN) difference between the model output and the reference sentence. The BoN training objective is differentiable and can be calculated efficiently without doing any approximations. Finally, we apply a three-stage training strategy to combine these two methods to train the NAT model. We validate our approach on four translation tasks (WMT14 En↔De, WMT16 En↔Ro), which shows that our approach largely outperforms NAT baselines and achieves remarkable performance on all translation tasks. The source code is available at https://github.com/ictnlp/Seq-NAT.

16 June 2023 in ACM Transactions on Asian and Low-Resource Language Information Processing
Verified
Journal Article
Open access
NYNian Yi
Nian Yi
CSChenze Shao
Chenze Shao
AWAishan Wumaier
Aishan Wumaier
Neural machine translation (NMT) mainly comprises the encoder and decoder. The encoder is mainly used to extract the feature vector of the source language sentence. The decoder predicts the next token according to the feature vector extracted by the encoder and the information of the current moment. In this process, there is no guarantee that the features extracted by the encoder are indistinguishable from the meaning of the sentences in the source language. There is also no guarantee that the decoder can accurately predict the corresponding character. These issues can lead to over-translation and under-translation issues in the translated results. Previous researchers alleviated this problem by calculating the gap between the reconstructed source-language sentences and the source-language sentences. Inspired by this method, we propose to integrate a reconstructor and a post-editor into NMT during the training. The reconstructor takes the translation of NMT as input to reconstruct the source sentence, and the post-editor takes the translation as input and post-edits it to predict the target sentence. Through the training of the reconstructor and the post-editor, the semantics of the translation are forced to follow the source sentence and the target sentence. Experimental results show that our approach can effectively improve the performance of NMT on multiple translation tasks.

18 March 2022 in arXiv
Verified
Preprint
Open access
PDF file
CSChenze Shao
Chenze Shao
Yan FengYan Feng
Yan Feng
Neural networks tend to gradually forget the previously learned knowledge when learning multiple tasks sequentially from dynamic data distributions. This problem is called \textit{catastrophic forgetting}, which is a fundamental challenge in the continual learning of neural networks. In this work, we observe that catastrophic forgetting not only occurs in continual learning but also affects the traditional static training. Neural networks, especially neural machine translation models, suffer from catastrophic forgetting even if they learn from a static training set. To be specific, the final model pays imbalanced attention to training samples, where recently exposed samples attract more attention than earlier samples. The underlying cause is that training samples do not get balanced training in each model update, so we name this problem \textit{imbalanced training}. To alleviate this problem, we propose Complementary Online Knowledge Distillation (COKD), which uses dynamically updated teacher models trained on specific data orders to iteratively provide complementary knowledge to the student model. Experimental results on multiple machine translation tasks show that our method successfully alleviates the problem of imbalanced training and achieves substantial improvements over strong baseline systems.

14 November 2023 in arXiv
Verified
Preprint
Open access
PDF file
SGShangtong Gui
Shangtong Gui
CSChenze Shao
Chenze Shao
ZMZhengrui Ma
Zhengrui Ma
XZXishan Zhang
Xishan Zhang
YCYunji Chen
Yunji Chen
Yan FengYan Feng
Yan Feng
Non-autoregressive Transformer(NAT) significantly accelerates the inference of neural machine translation. However, conventional NAT models suffer from limited expression power and performance degradation compared to autoregressive (AT) models due to the assumption of conditional independence among target tokens. To address these limitations, we propose a novel approach called PCFG-NAT, which leverages a specially designed Probabilistic Context-Free Grammar (PCFG) to enhance the ability of NAT models to capture complex dependencies among output tokens. Experimental results on major machine translation benchmarks demonstrate that PCFG-NAT further narrows the gap in translation quality between NAT and AT models. Moreover, PCFG-NAT facilitates a deeper understanding of the generated sentences, addressing the lack of satisfactory explainability in neural machine translation.Code is publicly available at https://github.com/ictnlp/PCFG-NAT.

1 September 2021 in arXiv
Verified
Preprint
Open access
PDF file
CSChenze Shao
Chenze Shao
Yan FengYan Feng
Yan Feng
JZJinchao Zhang
Jinchao Zhang
FMFandong Meng
Fandong Meng
JZJie Zhou
Jie Zhou
In recent years, Neural Machine Translation (NMT) has achieved notable results in various translation tasks. However, the word-by-word generation manner determined by the autoregressive mechanism leads to high translation latency of the NMT and restricts its low-latency applications. Non-Autoregressive Neural Machine Translation (NAT) removes the autoregressive mechanism and achieves significant decoding speedup through generating target words independently and simultaneously. Nevertheless, NAT still takes the word-level cross-entropy loss as the training objective, which is not optimal because the output of NAT cannot be properly evaluated due to the multimodality problem. In this article, we propose using sequence-level training objectives to train NAT models, which evaluate the NAT outputs as a whole and correlates well with the real translation quality. Firstly, we propose training NAT models to optimize sequence-level evaluation metrics (e.g., BLEU) based on several novel reinforcement algorithms customized for NAT, which outperforms the conventional method by reducing the variance of gradient estimation. Secondly, we introduce a novel training objective for NAT models, which aims to minimize the Bag-of-Ngrams (BoN) difference between the model output and the reference sentence. The BoN training objective is differentiable and can be calculated efficiently without doing any approximations. Finally, we apply a three-stage training strategy to combine these two methods to train the NAT model. We validate our approach on four translation tasks (WMT14 En$\leftrightarrow$De, WMT16 En$\leftrightarrow$Ro), which shows that our approach largely outperforms NAT baselines and achieves remarkable performance on all translation tasks. The source code is available at https://github.com/ictnlp/Seq-NAT.

22 June 2019 in arXiv
Verified
Preprint
Open access
PDF file
CSChenze Shao
Chenze Shao
Yan FengYan Feng
Yan Feng
JZJinchao Zhang
Jinchao Zhang
FMFandong Meng
Fandong Meng
XCXilin Chen
Xilin Chen
JZJie Zhou
Jie Zhou
Non-Autoregressive Transformer (NAT) aims to accelerate the Transformer model through discarding the autoregressive mechanism and generating target words independently, which fails to exploit the target sequential information. Over-translation and under-translation errors often occur for the above reason, especially in the long sentence translation scenario. In this paper, we propose two approaches to retrieve the target sequential information for NAT to enhance its translation ability while preserving the fast-decoding property. Firstly, we propose a sequence-level training method based on a novel reinforcement algorithm for NAT (Reinforce-NAT) to reduce the variance and stabilize the training procedure. Secondly, we propose an innovative Transformer decoder named FS-decoder to fuse the target sequential information into the top layer of the decoder. Experimental results on three translation tasks show that the Reinforce-NAT surpasses the baseline NAT system by a significant margin on BLEU without decelerating the decoding speed and the FS-decoder achieves comparable translation performance to the autoregressive Transformer with considerable speedup.