Deep learning has revolutionized pattern recognition, introducing tools that power a wide
range of technologies in such diverse fields as computer vision, natural language …

Abstract Recently, Sharpness-Aware Minimization (SAM), which connects the geometry of
the loss landscape and generalization, has demonstrated a significant performance boost …

Passing gradient is a widely used scheme in modern multi-node learning system (eg,
distributed training, collaborative learning). In a long time, people used to believe that …

DNN training is extremely time-consuming, necessitating efficient multi-accelerator
parallelization. Current approaches to parallelizing training primarily use intra-batch …

The vast majority of successful deep neural networks are trained using variants of stochastic
gradient descent (SGD) algorithms. Recent attempts to improve SGD can be broadly …

The great success of modern machine learning models on large datasets is contingent on
extensive computational resources with high financial and environmental costs. One way to …

Training large deep neural networks on massive datasets is computationally very
challenging. There has been recent surge in interest in using large batch stochastic …