Transformer is the state-of-the-art model in recent machine translation evaluations. Two
strands of research are promising to improve models of this kind: the first uses wide …

Unsupervised pre-training of large neural models has recently revolutionized Natural
Language Processing. By warm-starting from the publicly released checkpoints, NLP …

Large Transformer models yield impressive results on many tasks, but are expensive to
train, or even fine-tune, and so slow at decoding that their use and study becomes out of …

We explore the application of very deep Transformer models for Neural Machine Translation
(NMT). Using a simple yet effective initialization technique that stabilizes training, we show …

Translation quality evaluation plays a crucial role in machine translation. According to the
input format, it is mainly separated into three tasks, ie, reference-only, source-only and …

Fine-tuning pre-trained generative language models to down-stream language generation
tasks has shown promising results. However, this comes with the cost of having a single …

We present a simple and effective pretraining strategy--bidirectional training (BiT) for neural
machine translation. Specifically, we bidirectionally update the model parameters at the …

Multilingual neural machine translation (NMT), which translates multiple languages using a
single model, is of great practical importance due to its advantages in simplifying the training …

Self-attention based Transformer has demonstrated the state-of-the-art performances in a
number of natural language processing tasks. Self-attention is able to model long-term …

Non-autoregressive translation (NAT) models, which remove the dependence on previous
target tokens from the inputs of the decoder, achieve significantly inference speedup but at …