Fine-tuning pre-trained models has been ubiquitously proven to be effective in a wide range
of NLP tasks. However, fine-tuning the whole model is parameter inefficient as it always …

In the last year, new models and methods for pretraining and transfer learning have driven
striking performance improvements across a range of language understanding tasks. The …

This paper is a study of fine-tuning of BERT contextual representations, with focus on
commonly observed instabilities in few-sample scenarios. We identify several factors that …

Abstract We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP), a challenge set
for evaluating the linguistic knowledge of language models (LMs) on major grammatical …

We investigate how disagreement in natural language inference (NLI) annotation arises. We
developed a taxonomy of disagreement sources with 10 categories spanning 3 high-level …

Neural Architecture Search (NAS) is a promising and rapidly evolving research area.
Training a large number of neural networks requires an exceptional amount of …

Though state-of-the-art sentence representation models can perform tasks requiring
significant knowledge of grammar, it is an open question how best to evaluate their …

In recent years, a series of Transformer-based models unlocked major improvements in
general natural language understanding (NLU) tasks. Such a fast pace of research would …

We evaluate whether BERT, a widely used neural network for sentence processing,
acquires an inductive bias towards forming structural generalizations through pretraining on …

We study the settings for which deep contextual embeddings (eg, BERT) give large
improvements in performance relative to classic pretrained embeddings (eg, GloVe), and an …