Adam has been shown to outperform gradient descent on large language models by a
larger margin than on other tasks, but it is unclear why. We show that a key factor in this …

We propose Adam-mini, an optimizer that achieves on-par or better performance than
AdamW with 45% to 50% less memory footprint. Adam-mini reduces memory by cutting …

Adam is widely adopted in practical applications due to its fast convergence. However, its
theoretical analysis is still far from satisfactory. Existing convergence analyses for Adam rely …

Large language models (LLMs) offer personalized responses based on user interactions,
but this use case raises serious privacy concerns. Homomorphic encryption (HE) is a …

Large language models (LLMs) are notoriously memory-intensive during training,
particularly with the popular AdamW optimizer. This memory burden necessitates using …

In this work, we offer a theoretical analysis of two modern optimization techniques for
training large and complex models:(i) adaptive optimization algorithms, such as Adam, and …

In this work, we question the necessity of adaptive gradient methods for training deep neural
networks. SGD-SaI is a simple yet effective enhancement to stochastic gradient descent with …

Despite its widespread adoption, Adam's advantage over Stochastic Gradient Descent
(SGD) lacks a comprehensive theoretical explanation. This paper investigates Adam's …

Combining gradient compression methods (eg, CountSketch, quantization) and adaptive
optimizers (eg, Adam, AMSGrad) is a desirable goal in federated learning (FL), with potential …

Adaptive gradient methods have been widely adopted in training large-scale deep neural
networks, especially large foundation models. Despite the huge success in practice, their …