Media bias and the intolerance of media outlets and citizens to deal with opposing points of
view pose a threat to the proper functioning of democratic processes. In this respect, we …

Selecting a suitable pretraining dataset is crucial for both general-domain (eg, GPT-3) and
domain-specific (eg, Codex) language models (LMs). We formalize this problem as selecting …

Transformers-based models, such as BERT, have been one of the most successful deep
learning models for NLP. Unfortunately, one of their core limitations is the quadratic …

Language models pretrained on text from a wide variety of sources form the foundation of
today's NLP. In light of the success of these broad-coverage models, we investigate whether …

Transformer-based models are unable to process long sequences due to their self-attention
operation, which scales quadratically with the sequence length. To address this limitation …

Transformer-based models show their effectiveness across multiple domains and tasks. The
self-attention allows to combine information from all sequence elements into context-aware …

We propose pre-finetuning, an additional large-scale learning stage between language
model pre-training and fine-tuning. Pre-finetuning is massively multi-task learning (around …

The way the media presents events can significantly affect public perception, which in turn
can alter people's beliefs and views. Media bias describes a one-sided or polarizing …

We explore the task of predicting the leading political ideology or bias of news articles. First,
we collect and release a large dataset of 34,737 articles that were manually annotated for …

The increasing scale of general-purpose Pre-trained Language Models (PLMs) necessitates
the study of more efficient adaptation across different downstream tasks. In this paper, we …