GPUs are critical for maximizing the throughput-per-Watt of deep neural network (DNN)
applications. However, DNN applications often underutilize GPUs, even when using large …

Foundation models (eg, ChatGPT, DALL-E, PengCheng Mind, PanGu-) have demonstrated
extraordinary performance in key technological areas, such as natural language processing …

Bilevel optimization has been successfully applied to many important machine learning
problems. Algorithms for solving bilevel optimization have been studied under various …

Foundation models are in the process of becoming the dominant deep learning technology.
Pretraining a foundation model is always time-consuming due to the large scale of both the …

Integrated data analysis (IDA) pipelines---that combine data management (DM) and query
processing, high-performance computing (HPC), and machine learning (ML) training and …

Recent years have witnessed an exponential growth of model scale in deep learning-based
recommender systems---from Google's 2016 model with 1 billion parameters to the latest …

Decentralized algorithm is a form of computation that achieves a global goal through local
dynamics that relies on low-cost communication between directly-connected agents. On …

Communication compression is a crucial technique for modern distributed learning systems
to alleviate their communication bottlenecks over slower networks. Despite recent intensive …

Mixture of Expert (MoE) has received increasing attention for scaling DNN models to extra-
large size with negligible increases in computation. The MoE model has achieved the …

The transformer-based deep neural network (DNN) models have shown considerable
success across diverse tasks, prompting widespread adoption of distributed training …