This monograph surveys vision-language pre-training (VLP) methods for multimodal
intelligence that have been developed in the last few years. We group these approaches …

Modern natural language processing (NLP) methods employ self-supervised pretraining
objectives such as masked language modeling to boost the performance of various …

Large language models (LLMs) have shown remarkable capabilities in language
understanding and generation. However, such impressive capability typically comes with a …

Transformer models have achieved remarkable results in various natural language tasks,
but they are often prohibitively large, requiring massive memories and computational …

The increasing size of generative Pre-trained Language Models (PLMs) has greatly
increased the demand for model compression. Despite various methods to compress BERT …

Pre-trained Transformer-based models have achieved state-of-the-art performance for
various Natural Language Processing (NLP) tasks. However, these models often have …

Layer-wise distillation is a powerful tool to compress large models (ie teacher models) into
small ones (ie, student models). The student distills knowledge from the teacher by …

The primary goal of knowledge distillation (KD) is to encapsulate the information of a model
learned from a teacher network into a student network, with the latter being more compact …

Fine-grained classification involves dealing with datasets with larger number of classes with
subtle differences between them. Guiding the model to focus on differentiating dimensions …

Despite exciting progress in pre-training for visual-linguistic (VL) representations, very few
aspire to a small VL model. In this paper, we study knowledge distillation (KD) to effectively …