There is growing evidence of the effectiveness of Shampoo, a higher-order preconditioning
method, over Adam in deep learning optimization tasks. However, Shampoo's drawbacks …

Neural networks (NN) are extensively studied in cutting-edge soft sensor models due to their
feature extraction and function approximation capabilities. Current research into network …

We propose a novel approach to sequential Bayesian inference based on variational Bayes.
The key insight is that, in the online setting, we do not need to add the KL term to regularize …

Deep neural network ensembles are powerful tools for uncertainty quantification, which
have recently been re-interpreted from a Bayesian perspective. However, current methods …

Layer-wise preconditioning methods are a family of memory-efficient optimization algorithms
that introduce preconditioners per axis of each layer's weight tensors. These methods have …

First-order optimization methods are currently the mainstream in training deep neural
networks (DNNs). Optimizers like Adam incorporate limited curvature information by …

Structured large matrices are prevalent in machine learning. A particularly important class is
curvature matrices like the Hessian, which are central to understanding the loss landscape …

Many popular optimization methods can be united through fractional natural gradient
descent (FNGD), which pre-conditions the gradient with a fractional power of the inverse …