We showcase important features of the dynamics of the Stochastic Gradient Descent (SGD)
in the training of neural networks. We present empirical observations that commonly used …

Can we accelerate the convergence of gradient descent without changing the algorithm—
just by judiciously choosing stepsizes? Surprisingly, we show that the answer is yes. Our …

This work establishes new convergence guarantees for gradient descent in smooth convex
optimization via a computer-assisted analysis technique. Our theory allows nonconstant …

We develop a convergence-rate analysis of momentum with cyclical step-sizes. We show
that under some assumption on the spectral gap of Hessians in machine learning, cyclical …

Computing the Jacobian of the solution of an optimization problem is a central problem in
machine learning, with applications in hyperparameter optimization, meta-learning …

Understanding generalization in deep learning has been one of the major challenges in
statistical learning theory over the last decade. While recent work has illustrated that the …

Recent studies have shown that gradient descent (GD) can achieve improved generalization
when its dynamics exhibits a chaotic behavior. However, to obtain the desired effect, the …

Smart factories employ intelligent transportaton systems such as autonomous mobile robots
(AMRs) to support real-time adjusted production flows for agile and flexible production …

We conduct a comprehensive investigation into the dynamics of gradient descent using
large-order constant step-sizes in the context of quadratic regression models. Within this …

Why do neural networks trained with large learning rates for longer time often lead to better
generalization? In this paper, we delve into this question by examining the relation between …