We propose Adam-mini, an optimizer that achieves on par or better performance than
AdamW with 50% less memory footprint. Adam-mini reduces memory by cutting down the …

There is growing evidence of the effectiveness of Shampoo, a higher-order preconditioning
method, over Adam in deep learning optimization tasks. However, Shampoo's drawbacks …

AdamW has been the default optimizer for transformer pretraining. For many years, our
community searches for faster and more stable optimizers with only constraint positive …

The remarkable success of large language pretraining and the discovery of scaling laws
signify a paradigm shift in machine learning. Notably, the primary objective has evolved from …

Neural Information Retrieval has advanced rapidly in high-resource languages, but progress
in lower-resource ones such as Japanese has been hindered by data scarcity, among other …

Second-order optimizers, maintaining a matrix termed a preconditioner, are superior to first-
order optimizers in both theory and practice. The states forming the preconditioner and its …

Training large-scale models under given resources requires careful design of parallelism
strategies. In particular, the efficiency notion of critical batch size (CBS), concerning the …

We consider the problem of minimizing the average of a large number of smooth but
possibly non-convex functions. In the context of most machine learning applications, each …

This work investigates the effectiveness of schedule-free methods, developed by A. Defazio
et al.(NeurIPS 2024), in nonconvex optimization settings, inspired by their remarkable …

An AI tool has been developed to provide interpretable support for the diagnosis of BCC via
teledermatology, thus speeding up referrals and optimizing resource utilization. The …