Reciprocal teacher-student learning via forward and feedback knowledge distillation

J Gou, Y Chen, B Yu, J Liu, L Du… - IEEE transactions on …, 2024‏ - ieeexplore.ieee.org
Knowledge distillation (KD) is a prevalent model compression technique in deep learning,
aiming to leverage knowledge from a large teacher model to enhance the training of a …

Logit standardization in knowledge distillation

S Sun, W Ren, J Li, R Wang… - Proceedings of the IEEE …, 2024‏ - openaccess.thecvf.com
Abstract Knowledge distillation involves transferring soft labels from a teacher to a student
using a shared temperature-based softmax function. However the assumption of a shared …

Good teachers explain: Explanation-enhanced knowledge distillation

A Parchami-Araghi, M Böhle, S Rao… - European Conference on …, 2024‏ - Springer
Abstract Knowledge Distillation (KD) has proven effective for compressing large teacher
models into smaller student models. While it is well known that student models can achieve …

[HTML][HTML] A Survey on Knowledge Distillation: Recent Advancements

A Moslemi, A Briskina, Z Dang, J Li - Machine Learning with Applications, 2024‏ - Elsevier
Deep learning has achieved notable success across academia, medicine, and industry. Its
ability to identify complex patterns in large-scale data and to manage millions of parameters …

Efficient crowd counting via dual knowledge distillation

R Wang, Y Hao, L Hu, X Li, M Chen… - … on Image Processing, 2023‏ - ieeexplore.ieee.org
Most researchers focus on designing accurate crowd counting models with heavy
parameters and computations but ignore the resource burden during the model deployment …

Expanding and refining hybrid compressors for efficient object re-identification

Y **
G Yang, S Yu, H Yang, Z Nie, J Wang - Plos one, 2023‏ - journals.plos.org
Previous studies have shown that deep models are often over-parameterized, and this
parameter redundancy makes deep compression possible. The redundancy of model weight …

[HTML][HTML] Monocular depth estimation from a fisheye camera based on knowledge distillation

E Son, J Choi, J Song, Y **, SJ Lee - Sensors, 2023‏ - mdpi.com
Monocular depth estimation is a task aimed at predicting pixel-level distances from a single
RGB image. This task holds significance in various applications including autonomous …