In the past decade, advances in deep learning have resulted in breakthroughs in a variety of
areas, including computer vision, natural language understanding, speech recognition, and …

Transformer-based Deep Neural Network architectures have gained tremendous interest
due to their effectiveness in various applications across Natural Language Processing (NLP) …

Abstract Knowledge tracing (KT) aims to trace students' knowledge states by predicting
whether students answer correctly on exercises. Despite the excellent performance of …

Neural Architecture Search (NAS) is widely used to automatically obtain the neural network
with the best performance among a large number of candidate architectures. To reduce the …

Abstract Neural Architecture Search (NAS) has shown promising performance in the
automatic design of vision transformers (ViT) exceeding 1G FLOPs. However, designing …

Supernet is a core component in many recent Neural Architecture Search (NAS) methods. It
not only helps embody the search space but also provides a (relative) estimation of the final …

The success of multi-task learning (MTL) can largely be attributed to the shared
representation of related tasks, allowing the models to better generalise. In deep learning …

Based on the weight-sharing mechanism, one-shot NAS methods train a supernet and then
inherit the pre-trained weights to evaluate sub-models, largely reducing the search cost …

Efficient detectors for edge devices are often optimized for parameters or speed count
metrics, which remain in weak correlation with the energy of detectors. However, some …

In this paper, we propose Stochastic Knowledge Distillation (SKD) to obtain compact BERT-
style language model dubbed SKDBERT. In each distillation iteration, SKD samples a …