Model compression for deep neural networks: A survey
Currently, with the rapid development of deep learning, deep neural networks (DNNs) have
been widely applied in various computer vision tasks. However, in the pursuit of …
been widely applied in various computer vision tasks. However, in the pursuit of …
Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks
Deep neural models, in recent years, have been successful in almost every field, even
solving the most complex problem statements. However, these models are huge in size with …
solving the most complex problem statements. However, these models are huge in size with …
Knowledge distillation: A survey
In recent years, deep neural networks have been successful in both industry and academia,
especially for computer vision tasks. The great success of deep learning is mainly due to its …
especially for computer vision tasks. The great success of deep learning is mainly due to its …
Knowledge distillation meets self-supervision
Abstract Knowledge distillation, which involves extracting the “dark knowledge” from a
teacher network to guide the learning of a student network, has emerged as an important …
teacher network to guide the learning of a student network, has emerged as an important …
Exploring inter-channel correlation for diversity-preserved knowledge distillation
Abstract Knowledge Distillation has shown very promising ability in transferring learned
representation from the larger model (teacher) to the smaller one (student). Despite many …
representation from the larger model (teacher) to the smaller one (student). Despite many …
A survey on green deep learning
In recent years, larger and deeper models are springing up and continuously pushing state-
of-the-art (SOTA) results across various fields like natural language processing (NLP) and …
of-the-art (SOTA) results across various fields like natural language processing (NLP) and …
[PDF][PDF] Knowledge distillation via softmax regression representation learning
J Yang, B Martinez, A Bulat, G Tzimiropoulos - 2021 - qmro.qmul.ac.uk
This paper addresses the problem of model compression via knowledge distillation. We
advocate for a method that optimizes the output feature of the penultimate layer of the …
advocate for a method that optimizes the output feature of the penultimate layer of the …
R-dfcil: Relation-guided representation learning for data-free class incremental learning
Abstract Class-Incremental Learning (CIL) struggles with catastrophic forgetting when
learning new knowledge, and Data-Free CIL (DFCIL) is even more challenging without …
learning new knowledge, and Data-Free CIL (DFCIL) is even more challenging without …
Unpaired multi-modal segmentation via knowledge distillation
Multi-modal learning is typically performed with network architectures containing modality-
specific layers and shared layers, utilizing co-registered images of different modalities. We …
specific layers and shared layers, utilizing co-registered images of different modalities. We …
More grounded image captioning by distilling image-text matching model
Visual attention not only improves the performance of image captioners, but also serves as a
visual interpretation to qualitatively measure the caption rationality and model transparency …
visual interpretation to qualitatively measure the caption rationality and model transparency …