In practical scenarios, it is necessary to build variable-sized models to accommodate diverse
resource constraints, where weight initialization serves as a crucial step preceding training …

Many applications today provide users with multiple auto-complete drafts as they type,
including GitHub's code completion, Gmail's smart compose, and Apple's messaging auto …

This paper introduces a new learning paradigm termed Neural Metamorphosis (NeuMeta),
which aims to build self-morphable neural networks. Contrary to crafting separate models for …

Recent developments in large language models have sparked interest in efficient
pretraining methods. Stagewise training approaches to improve efficiency, like gradual …

Abstract Knowledge distillation is commonly used to compress an ensemble of models into a
single model. In this work we study the problem of progressive ensemble distillation: Given a …

Effective approaches that can scale embedding model depth (ie layers) and embedding size
allow for the creation of models that are highly scalable across different computational …

State Space Models (SSMs) like Mamba2 are a promising alternative to Transformers, with
faster theoretical training and inference times--especially for long context lengths. Recent …

Web-scale search systems learn an encoder to embed a given query which is then hooked
into an approximate nearest neighbor search (ANNS) pipeline to retrieve similar data points …

This position paper argues that in many realistic (ie, complex, contextualized, subjective)
scenarios, one LLM is not enough to produce a reliable output. We challenge the status quo …

Training large language models (LLMs) for different inference constraints is computationally
expensive, limiting control over efficiency-accuracy trade-offs. Moreover, once trained, these …