Recent work has demonstrated that increased training dataset diversity improves general
cross-domain knowledge and downstream generalization capability for large-scale …

Contrastive representation learning has been outstandingly successful in practice. In this
work, we identify two key properties related to the contrastive loss:(1) alignment (closeness) …

Large language models (LLMs) based on transformer are witnessing a notable trend of size
expansion, which brings considerable costs to both model training and inference. However …

This datasheet describes the Pile, a 825 GiB dataset of human-authored text compiled by
EleutherAI for use in large-scale language modeling. The Pile is comprised of 22 different …

Recent literature has underscored the importance of dataset documentation work for
machine learning, and part of this work involves addressing" documentation debt" for …

We use a dataset of US first names with labels based on predominant gender and racial
group to examine the effect of training corpus frequency on tokenization, contextualization …

This paper contributes a formal case study in retrospective dataset documentation and
pinpoints several problems with the influential BookCorpus dataset. Recent work has …

Memorization in large language models (LLMs) is a growing concern. LLMs have been
shown to easily reproduce parts of their training data, including copyrighted work. This is an …

Leveraging pre-trained language models (PLMs) as initializers for efficient transfer learning
has become a universal approach for text-related tasks. However, the models not only learn …

We present LogiGAN, an unsupervised adversarial pre-training framework for improving
logical reasoning abilities of language models. Upon automatic identification of logical …