Successful integration of deep neural networks (DNNs) or deep learning (DL) has resulted
in breakthroughs in many areas. However, deploying these highly accurate models for data …

Expanding the language coverage of speech technology has the potential to improve
access to information for many more people. However, current speech technology is …

This paper presents XLS-R, a large-scale model for cross-lingual speech representation
learning based on wav2vec 2.0. We train models with up to 2B parameters on nearly half a …

Despite their wide adoption, the underlying training and memorization dynamics of very
large language models is not well understood. We empirically study exact memorization in …

Generative language models define distributions over sequences of tokens that can
represent essentially any combination of data modalities (eg, any permutation of image …

Scaling up training datasets and model parameters have benefited neural network-based
language models, but also present challenges like distributed compute, input data …

We introduce CM3, a family of causally masked generative models trained over a large
corpus of structured multi-modal documents that can contain both text and image tokens …

The success of Transformer models has pushed the deep learning model scale to billions of
parameters, but the memory limitation of a single GPU has led to an urgent need for training …

Transformer based large language models have achieved tremendous success. However,
the significant memory and computational costs incurred during the inference process make …

Transformer models have achieved state-of-the-art performance on various domains of
applications and gradually becomes the foundations of the advanced large deep learning …