Abstract Knowledge distillation (KD) has been widely used to improve the test accuracy of a"
student" network, by training it to mimic the soft probabilities of a trained" teacher" network …

Despite the empirical success and practical significance of (relational) knowledge distillation
that matches (the relations of) features between teacher and student models, the …

Knowledge distillation (KD) compresses deep neural networks by transferring task-related
knowledge from cumbersome pre-trained teacher models to compact student models …

A primary challenge in large language model (LLM) development is their onerous pre-
training cost. Typically, such pre-training involves optimizing a self-supervised objective …

We characterize the statistical efficiency of knowledge transfer through $ n $ samples from a
teacher to a probabilistic student classifier with input space $\mathcal S $ over labels …

The rising footprint of machine learning has led to a focus on imposing model sparsity as a
means of reducing computational and memory costs. For deep neural networks (DNNs), the …

A core component present in many successful neural network architectures, is an MLP block
of two fully connected layers with a non-linear activation in between. An intriguing …

Knowledge distillation leverages a teacher model to improve the training of a student model.
A persistent challenge is that a better teacher does not always yield a better student, to …

We provide a distillation scaling law that estimates distilled model performance based on a
compute budget and its allocation between the student and teacher. Our findings reduce the …

Despite the popularity and success of deep learning, there is limited understanding of when,
how, and why neural networks generalize to unseen examples. Since learning can be seen …