Probing classifiers have emerged as one of the prominent methodologies for interpreting
and analyzing deep neural network models of natural language processing. The basic idea …

A possible explanation for the impressive performance of masked language model (MLM)
pre-training is that such models have learned to represent the syntactic structures prevalent …

Does the effectiveness of neural language models derive entirely from accurate modeling of
surface word co-occurrence statistics, or do these models represent and reason about the …

NLP is currently dominated by general-purpose pretrained language models like RoBERTa,
which achieve strong performance on NLU tasks through pretraining on billions of words …

In the last half-decade, the field of natural language processing (NLP) has undergone two
major transitions: the switch to neural networks as the primary modeling paradigm and the …

Pretrained language models have been shown to encode relational information, such as the
relations between entities or concepts in knowledge-bases--(Paris, Capital, France) …

Most interpretability research in NLP focuses on understanding the behavior and features of
a fully trained model. However, certain insights into model behavior may only be accessible …

Recent studies have shown that language models pretrained and/or fine-tuned on randomly
permuted sentences exhibit competitive performance on GLUE, putting into question the …

A central quest of probing is to uncover how pre-trained models encode a linguistic property
within their representations. An encoding, however, might be spurious-ie, the model might …

This chapter critically examines the potential contributions of modern language models to
theoretical linguistics. Despite their focus on engineering goals, these models' ability to …