Recently, with the rapid developments of the Internet and social networks, there have been
tremendous increase in the amount of complex-structured text resources. These information …

We study the design decision of publicly available instruction tuning methods, by
reproducing and breaking down the development of Flan 2022 (Chung et al., 2022) …

There has been growing interest in parameter-efficient methods to apply pre-trained
language models to downstream tasks. Building on the Prompt Tuning approach of Lester et …

Averaging the parameters of models that have the same architecture and initialization can
provide a means of combining their respective capabilities. In this paper, we take the …

Despite the recent success of multi-task learning and transfer learning for natural language
processing (NLP), few works have systematically studied the effect of scaling up the number …

Large language models have recently been shown to attain reasonable zero-shot
generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that …

Prompt tuning (PT) is a promising parameter-efficient method to utilize extremely large pre-
trained language models (PLMs), which can achieve comparable performance to full …

Recent work has shown that fine-tuning large pre-trained language models on a collection
of tasks described via instructions, aka instruction-tuning, improves their zero and few-shot …

Prompt tuning, in which a base pretrained model is adapted to each task via conditioning on
learned prompt vectors, has emerged as a promising approach for efficiently adapting large …

Humans can learn a new language task efficiently with only few examples, by leveraging
their knowledge obtained when learning prior tasks. In this paper, we explore whether and …