This article surveys and organizes research works in a new paradigm in natural language
processing, which we dub “prompt-based learning.” Unlike traditional supervised learning …

The field of machine translation (MT), the automatic translation of written text from one
natural language into another, has experienced a major paradigm shift in recent years …

We present BART, a denoising autoencoder for pretraining sequence-to-sequence models.
BART is trained by (1) corrupting text with an arbitrary noising function, and (2) learning a …

While large-scale language models (LMs) are able to imitate the distribution of natural
language well enough to generate realistic text, it is difficult to control which regions of the …

This paper presents a new sequence-to-sequence pre-training model called ProphetNet,
which introduces a novel self-supervised objective named future n-gram prediction and the …

Neural text generation is a key tool in natural language applications, but it is well known
there are major problems at its core. In particular, standard likelihood training and decoding …

We introduce extreme summarization, a new single-document summarization task which
does not favor extractive strategies and calls for an abstractive modeling approach. The idea …

In open-domain dialogue intelligent agents should exhibit the use of knowledge, however
there are few convincing demonstrations of this to date. The most popular sequence to …

Neural abstractive summarization models are flexible and can produce coherent summaries,
but they are sometimes unfaithful and can be difficult to control. While previous studies …

We introduce the first large-scale corpus for long-form question answering, a task requiring
elaborate and in-depth answers to open-ended questions. The dataset comprises 270K …