Deep learning, which is a subfield of machine learning, has opened a new era for the
development of neural networks. The auto-encoder is a key component of deep structure …

In the age of artificial intelligence, the best approach to handling huge amounts of data is a
tremendously motivating and hard problem. Among machine learning models, stochastic …

SGD performs worse than Adam by a significant margin on Transformers, but the reason
remains unclear. In this work, we provide an explanation through the lens of Hessian:(i) …

The monkeypox virus poses a new pandemic threat while we are still recovering from
COVID-19. Despite the fact that monkeypox is not as lethal and contagious as COVID-19 …

Scaling up the convolutional neural network (CNN) size (eg, width, depth, etc.) is known to
effectively improve model accuracy. However, the large model size impedes training on …

In this paper, we provide a rigorous proof of convergence of the Adaptive Moment Estimate
(Adam) algorithm for a wide class of optimization objectives. Despite the popularity and …

Facial emotion recognition (FER) is significant for human-computer interaction such as
clinical practice and behavioral description. Accurate and robust FER by computer models …

Whether you deal with a real-life issue or create a software product, optimization is
constantly the ultimate goal. This goal, however, is achieved by utilizing one of the …

Ever since\citet {reddi2019convergence} pointed out the divergence issue of Adam, many
new variants have been designed to obtain convergence. However, vanilla Adam remains …

Selecting an optimizer is a central step in the contemporary deep learning pipeline. In this
paper, we demonstrate the sensitivity of optimizer comparisons to the hyperparameter tuning …