With the recent advances in deep learning, different approaches to improving pre-trained
language models (PLMs) have been proposed. PLMs have advanced state-of-the-art …

Transformer plays a vital role in the realms of natural language processing (NLP) and
computer vision (CV), specially for constructing large language models (LLM) and large …

Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task has been a
well-known paradigm in Natural Language Processing. However, with the ever-growing size …

The burgeoning field of Large Language Models (LLMs), exemplified by sophisticated
models like OpenAI's ChatGPT, represents a significant advancement in artificial …

Recent advances in self-supervised learning and the Transformer architecture have
significantly improved natural language processing (NLP), achieving remarkably low …

The research field of Information Retrieval (IR) has evolved significantly, expanding beyond
traditional search to meet diverse user information needs. Recently, Large Language …

In computer vision, it has achieved great transfer learning performance via adapting large-
scale pretrained vision models (eg, vision transformers) to downstream tasks. Common …

The increasing size of generative Pre-trained Language Models (PLMs) has greatly
increased the demand for model compression. Despite various methods to compress BERT …

Despite achieving state-of-the-art performance on many NLP tasks, the high energy cost and
long inference delay prevent Transformer-based pretrained language models (PLMs) from …

Auto-regressive large language models such as GPT-3 require enormous computational
resources to use. Traditionally, structured pruning methods are employed to reduce …