Unrolled computation graphs arise in many scenarios, including training RNNs, tuning
hyperparameters through unrolled optimization, and training learned optimizers. Current …

Abstract We introduce Meta-AdaM, a meta-learned adaptive optimizer with momentum,
designed for few-shot learning tasks that pose significant challenges to deep learning …

Offline model-based optimization aims to maximize a black-box objective function with a
static dataset of designs and their scores. In this paper, we focus on biological sequence …

This paper proposes a new hierarchical approach to learning rate adaptation in gradient
methods, called learning rate optimization (LRO). LRO formulates the learning rate adaption …

The behaviors of deep neural networks (DNNs) are notoriously resistant to human
interpretations. In this paper, we propose Hypergradient Data Relevance Analysis, or …

The choice of learning rate (LR) functions and policies has evolved from a simple fixed LR to
the decaying LR and the cyclic LR, aiming to improve the accuracy and reduce the training …

ABSTRACT Inertial Measurement Units (IMU) are commonly used in inertial attitude
estimation from engineering to medical sciences. There may be disturbances and high …

We propose a framework for online meta-optimization of parameters that govern
optimization, called Amortized Proximal Optimization (APO). We first interpret various …

This paper presents a novel end-to-end deep learning framework for real-time inertial
attitude estimation using 6DoF IMU measurements. Inertial Measurement Units are widely …

Research into optimisation for deep learning is characterised by a tension between the
computational efficiency of first-order, gradient-based methods (such as SGD and Adam) …