Model merging is an efficient empowerment technique in the machine learning community
that does not require the collection of raw training data and does not require expensive …

Mixture-of-experts (MoE) is gaining increasing attention due to its unique properties and
remarkable performance, especially for language tasks. By sparsely activating a subset of …

Low-Rank Adaptation (LoRA) is a popular technique for parameter-efficient fine-tuning of
Large Language Models (LLMs). We study how different LoRA modules can be merged to …

Training high-performing large language models (LLMs) from scratch is a notoriously
expensive and difficult task, costing hundreds of millions of dollars in compute alone. These …

By increasing model parameters but activating them sparsely when performing a task, the
use of Mixture-of-Experts (MoE) architecture significantly improves the performance of Large …

Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption
and redundancy in experts. Pruning MoE can reduce network weights while maintaining …

The emergence of Mixture of Experts (MoE) LLMs has significantly advanced the
development of language models. Compared to traditional LLMs, MoE LLMs outperform …

The emergence of large-scale Mixture of Experts (MoE) models has marked a significant
advancement in artificial intelligence, offering enhanced model capacity and computational …

The Mixture-of-Experts (MoE) architecture has demonstrated significant advantages in the
era of Large Language Models (LLMs), offering enhanced capabilities with reduced …

Sparse Mixture of Expert (SMoE) models have emerged as a scalable alternative to dense
models in language modeling. These models use conditionally activated feedforward …