Transformer-based models have pushed state of the art in many areas of NLP, but our
understanding of what is behind their success is still limited. This paper is the first survey of …

Transformer architectures have proven to learn useful representations for protein
classification and generation tasks. However, these representations present challenges in …

Recently, many pre-trained language models for source code have been proposed to model
the context of code and serve as a basis for downstream code intelligence tasks such as …

The success of large pretrained language models (LMs) such as BERT and RoBERTa has
sparked interest in probing their representations, in order to unveil what types of knowledge …

Vector-based word representation paradigms situate lexical meaning at different levels of
abstraction. Distributional and static embedding models generate a single vector per word …

Distributional semantics has deeply changed in the last decades. First, predict models stole
the thunder from traditional count ones, and more recently both of them were replaced in …

Static word embeddings that represent words by a single vector cannot capture the
variability of word meaning in different linguistic and extralinguistic contexts. Building on …

Multiword expressions (MWEs) are composed of multiple words and exhibit variable
degrees of compositionality. As such, their meanings are notoriously difficult to model, and it …

This paper presents MedRoBERTa. nl as the first Transformer-based language model for
Dutch medical language. We show that using 13GB of text data from Dutch hospital notes …

The advent of contextual word embeddings--representations of words which incorporate
semantic and syntactic information from their context--has led to tremendous improvements …