Abstract Low-Rank Adaptation (LoRA), which updates the dense neural network layers with
pluggable low-rank matrices, is one of the best performed parameter efficient fine-tuning …

In this paper, we study the role of initialization in Low Rank Adaptation (LoRA) as originally
introduced in Hu et al.(2021). Essentially, to start from the pretrained model, one can either …

Current PEFT methods for LLMs can achieve high quality, efficient training, or scalable
serving, but not all three simultaneously. To address this limitation, we investigate sparse …

Recent efforts to scale Transformer models have demonstrated rapid progress across a wide
range of tasks (Wei at. al 2022). However, fine-tuning these models for downstream tasks is …

Leveraging large language models (LLMs) for complex natural language tasks typically
requires long-form prompts to convey detailed requirements and information, which results …

Scaling language models to handle longer input sequences typically necessitates large key-
value (KV) caches, resulting in substantial memory overhead during inference. In this paper …

Low-rank adaptation (LoRA) and its variants have recently gained much interest due to their
ability to avoid excessive inference costs. However, LoRA still encounters the following …

Although many efforts have been made, it is still a challenge to balance the training budget,
downstream performance, and the general capabilities of the LLMs in many applications …

Large Language Models (LLMs) have demonstrated impressive performance across various
domains. However, the enormous number of model parameters makes fine-tuning …

Large language models (LLMs) have shown substantial progress in natural language
understanding and generation, proving valuable especially in the medical field. Despite …