Transformer language models have received widespread public attention, yet their
generated text is often surprising even to NLP researchers. In this survey, we discuss over …

The field of natural language processing has seen impressive progress in recent years, with
neural network models replacing many of the traditional systems. A plethora of new models …

Language models demonstrate both quantitative improvement and new qualitative
capabilities with increasing scale. Despite their potentially transformative impact, these new …

Rapid progress in machine learning for natural language processing has the potential to
transform debates about how humans learn language. However, the learning environments …

A machine learning system can score well on a given test set by relying on heuristics that
are effective for frequent example types but break down in more challenging cases. We …

Modern deep neural networks achieve impressive performance in engineering applications
that require extensive linguistic skills, such as machine translation. This success has …

How and to what extent does BERT encode syntactically-sensitive hierarchical information
or positionally-sensitive linear information? Recent work has shown that contextual …

RNN language models have achieved state-of-the-art perplexity results and have proven
useful in a suite of NLP tasks, but it is as yet unclear what syntactic generalizations they …

One reason pretraining on self-supervised linguistic tasks is effective is that it teaches
models features that are helpful for language understanding. However, we want pretrained …

If the same neural network architecture is trained multiple times on the same dataset, will it
make similar linguistic generalizations across runs? To study this question, we fine-tuned …