It has long been argued that only humans could produce and understand language. But
now, for the first time, artificial language models (LMs) achieve this feat. Here we survey the …

Transformer-based models have pushed state of the art in many areas of NLP, but our
understanding of what is behind their success is still limited. This paper is the first survey of …

Transformer architectures have proven to learn useful representations for protein
classification and generation tasks. However, these representations present challenges in …

Does the effectiveness of neural language models derive entirely from accurate modeling of
surface word co-occurrence statistics, or do these models represent and reason about the …

In natural language processing, short-text semantic similarity (STSS) is a very prominent
field. It has a significant impact on a broad range of applications, such as question …

Abstract Machines have achieved a broad and growing set of linguistic competencies,
thanks to recent progress in Natural Language Processing (NLP). Psychologists have …

Pre-trained language models (LMs) encode rich information about linguistic structure but
their knowledge about lexical polysemy remains unclear. We propose a novel experimental …

Distributional semantics has deeply changed in the last decades. First, predict models stole
the thunder from traditional count ones, and more recently both of them were replaced in …

Clustering token-level contextualized word representations produces output that shares
many similarities with topic models for English text collections. Unlike clusterings of …

Transformer-based language models have taken many fields in NLP by storm. BERT and its
derivatives dominate most of the existing evaluation benchmarks, including those for Word …