Abstract Knowledge Distillation (KD) uses the teacher's prediction logits as soft labels to
guide the student, while self-KD does not need a real teacher to require the soft labels. This …

Abstract Existing Knowledge Distillation (KD) methods typically focus on transferring
knowledge from a large-capacity teacher to a low-capacity student model achieving …

Deep neural networks tend to suffer from the overfitting issue when the training data are not
enough. In this paper, we introduce two metrics from the intra-class distribution of correct …

Abstract Self-Knowledge Distillation (Self-KD), a technique that enables neural networks to
learn from themselves, often relies on auxiliary modules or networks to generate supervisory …

Class incremental learning (CIL) enables deep networks to progressively learn new tasks
while remembering previously learned knowledge. A popular design for CIL involves …

Soft pseudo-labels, generated by the softmax predictions of the trained networks, offer a
probabilistic rather than binary form, and have been shown to improve the performance of …

Spiking neural networks (SNNs) have attracted considerable attention for their event-driven,
low-power characteristics and high biological interpretability. Inspired by knowledge …

Since pioneering work of Hinton et al., knowledge distillation based on Kullback-Leibler
Divergence (KL-Div) has been predominant, and recently its variants have achieved …

To fully exploit knowledge from self-knowledge distillation network in which a student model
is progressively trained to distill its own knowledge without a pre-trained teacher model, a …

We present a novel adversarial penalized self-knowledge distillation method, named
adversarial learning and implicit regularization for self-knowledge distillation (AI-KD), which …