Large multilingual language models typically rely on a single vocabulary shared across
100+ languages. As these models have increased in parameter count and depth …

We present a novel way of injecting factual knowledge about entities into the pretrained
BERT model (Devlin et al., 2019): We align Wikipedia2Vec entity vectors (Yamada et al …

We introduce exBERT, a training method to extend BERT pre-trained models from a general
domain to a new pre-trained model for a specific domain with a new additive vocabulary …

Large pre-trained models such as BERT are known to improve different downstream NLP
tasks, even when such a model is trained on a generic domain. Moreover, recent studies …

Using model weights pretrained on a high-resource language as a warm start can reduce
the need for data and compute to obtain high-quality language models for other, especially …

The content on the web is in a constant state of flux. New entities, issues, and ideas
continuously emerge, while the semantics of the existing conversation topics gradually shift …

Domain adaptation of Pretrained Language Models (PTLMs) is typically achieved by
unsupervised pretraining on target-domain text. While successful, this approach is …

The rapid development of social networks, electronic commerce, mobile Internet, and other
technologies, has influenced the growth of Web data. Social media and Internet forums are …

Instead of pretraining multilingual language models from scratch, a more efficient method is
to adapt existing pretrained language models (PLMs) to new languages via vocabulary …

Recent research analyzing the sensitivity of natural language understanding models to word-
order perturbations has shown that neural models are surprisingly insensitive to the order of …