Self-distillation (SD) is the process of first training a" teacher" model and then using its
predictions to train a" student" model that has the same architecture. Specifically, the …

Neural Architecture Search (NAS) methods are widely used in various industries to obtain
high-quality, task-specific solutions with minimal human intervention. Event Sequences …

Abstract Knowledge distillation (KD) has been widely used to improve the test accuracy of a"
student" network, by training it to mimic the soft probabilities of a trained" teacher" network …

We consider scenarios where a very accurate (often small) predictive model using restricted
features is available when training a full-featured (often larger) model. This restricted model …

Abstract Knowledge distillation is a commonly-used compression method in ML due to the
popularity of increasingly large-scale models, but it is unclear if all the information a teacher …

Low-rank adapters (LoRA) and their variants are popular parameter-efficient finetuning
(PEFT) techniques that closely match full model fine-tune performance while requiring only a …

Low-rank adapters (LoRA) and their variants are popular parameter-efficient fine-tuning
(PEFT) techniques that closely match full model fine-tune performance while requiring only a …

Bayesian optimization (BO) has contributed greatly to improving model performance by
suggesting promising hyperparameter configurations iteratively based on observations from …

The rising popularity of large foundation models has led to a heightened demand for
parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), which offer …

The traditional three-stage pruning pipeline is first to train an original dense network, then
identify redundant parts of the network for pruning based on the evaluation metrics of the …