Deep learning has recently come to dominate computational linguistics, leading to claims of
human-level performance in a range of language processing tasks. Like much previous …

A possible explanation for the impressive performance of masked language model (MLM)
pre-training is that such models have learned to represent the syntactic structures prevalent …

Pre-training by language modeling has become a popular and successful approach to NLP
tasks, but we have yet to understand exactly what linguistic capacities these pre-training …

Contextual word representations derived from large-scale neural language models are
successful across a diverse set of NLP tasks, suggesting that they encode useful and …

Graph neural networks (GNNs) have become a popular approach to integrating structural
inductive biases into NLP models. However, there has been little work on interpreting them …

Abstract We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP), a challenge set
for evaluating the linguistic knowledge of language models (LMs) on major grammatical …

The ability to generalise well is one of the primary desiderata of natural language
processing (NLP). Yet, what'good generalisation'entails and how it should be evaluated is …

As the use of deep learning techniques has grown across various fields over the past
decade, complaints about the opaqueness of the black-box models have increased …

Recent work has shown that LSTMs trained on a generic language modeling objective
capture syntax-sensitive generalizations such as long-distance number agreement. We …

Negation is a core construction in natural language. Despite being very successful on many
tasks, state-of-the-art pre-trained language models often handle negation incorrectly. To …