Abstract Low-Rank Adaptation (LoRA), which updates the dense neural network layers with
pluggable low-rank matrices, is one of the best performed parameter efficient fine-tuning …

We introduce Holmes, a new benchmark designed to assess language models'(LMs')
linguistic competence—their unconscious understanding of linguistic phenomena …

Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption
and redundancy in experts. Pruning MoE can reduce network weights while maintaining …

Fine-tuning large language models (LLMs) with low-rank adaptations (LoRAs) has become
common practice, often yielding numerous copies of the same LLM differing only in their …

Parameter-efficient fine-tuning optimizes large, pre-trained foundation models by updating a
subset of parameters; in this class, Low-Rank Adaptation (LoRA) is particularly effective …

Breakthroughs in natural language processing (NLP) by large-scale language models
(LLMs) have led to superior performance in multilingual tasks such as translation …

Low-rank adaptation (LoRA) is a natural method for finetuning in communication-
constrained machine learning settings such as cross-device federated learning. Prior work …

With the growth of model sizes and scale of their deployment, their sheer size burdens the
infrastructure requiring more network and more storage to accommodate these. While there …

When language models (LMs) are trained to``unlearn''a skill, does this unlearning
generalize? We study the behavior of LMs after fine-tuned on data for a target task (eg …

As machine learning models grow in size and complexity, their maintainability becomes a
critical concern, especially when they are increasingly deployed in dynamic, real-world …