Transformer-based pretrained language models (T-PTLMs) have achieved great success in
almost every NLP task. The evolution of these models started with GPT and BERT. These …

Recent advances in transformer-based architectures have shown promise in several
machine learning tasks. In the audio domain, such architectures have been successfully …

The growing size of neural language models has led to increased attention in model
compression. The two predominant approaches are pruning, which gradually removes …

Pre-trained language models (eg, BERT (Devlin et al., 2018) and its variants) have achieved
remarkable success in varieties of NLP tasks. However, these models usually consist of …

Language model pre-training, such as BERT, has significantly improved the performances of
many natural language processing tasks. However, pre-trained language models are …

Pre-training has improved model accuracy for both classification and generation tasks at the
cost of introducing much larger and slower models. Pruning methods have proven to be an …

Magnitude pruning is a widely used strategy for reducing model size in pure supervised
learning; however, it is less effective in the transfer learning regime that has become …

While task-specific finetuning of pretrained networks has led to significant empirical
advances in NLP, the large size of networks makes finetuning difficult to deploy in multi-task …

Recent advances in state-of-the-art DNN architecture design have been moving toward
Transformer models. These models achieve superior accuracy across a wide range of …

Transformer-based language models have become a key building block for natural
language processing. While these models are extremely accurate, they can be too large and …