Reverse-mode automatic differentiation and optimization of GPU kernels via Enzyme

WS Moses, V Churavy, L Paehler… - Proceedings of the …, 2021 - dl.acm.org
Computing derivatives is key to many algorithms in scientific computing and machine
learning such as optimization, uncertainty quantification, and stability analysis. Enzyme is a …

An introduction to algorithmic differentiation

AH Gebremedhin, A Walther - Wiley Interdisciplinary Reviews …, 2020 - Wiley Online Library
Algorithmic differentiation (AD), also known as automatic differentiation, is a technology for
accurate and efficient evaluation of derivatives of a function given as a computer model. The …

Optimal memory-aware backpropagation of deep join networks

O Beaumont, J Herrmann… - … Transactions of the …, 2020 - royalsocietypublishing.org
Deep learning training memory needs can prevent the user from considering large models
and large batch sizes. In this work, we propose to use techniques from memory-aware …

Optimal checkpointing for heterogeneous chains: how to train deep neural networks with limited memory

J Herrmann, O Beaumont, L Eyraud-Dubois… - ar** schemes
H Zhang, EM Constantinescu - Journal of Computational Science, 2023 - Elsevier
We consider checkpointing strategies that minimize the number of recomputations needed
when performing discrete adjoint computations using multistage time-step** schemes that …

Asynchronous two-level checkpointing scheme for large-scale adjoints in the spectral-element solver Nek5000

M Schanen, O Marin, H Zhang, M Anitescu - Procedia Computer Science, 2016 - Elsevier
Adjoints are an important computational tool for large-scale sensitivity evaluation,
uncertainty quantification, and derivative-based optimization. An essential component of …

Bathymetric influences on Antarctic Ice‐Shelf melt rates

DN Goldberg, TA Smith… - Journal of …, 2020 - Wiley Online Library
Ocean bathymetry exerts a strong control on ice sheet‐ocean interactions within Antarctic
ice‐shelf cavities, where it can limit the access of warm, dense water at depth to the …

Reducing memory requirements of quantum optimal control

SHK Narayanan, T Propson, M Bongarti… - International Conference …, 2022 - Springer
Quantum optimal control problems are typically solved by gradient-based algorithms such
as GRAPE, which suffer from exponential growth in storage with increasing number of qubits …

Optimal Re-Materialization Strategies for Heterogeneous Chains: How to Train Deep Neural Networks with Limited Memory

O Beaumont, L Eyraud-Dubois, J Herrmann… - ACM Transactions on …, 2024 - dl.acm.org
Training in Feed Forward Deep Neural Networks is a memory-intensive operation which is
usually performed on GPUs with limited memory capacities. This may force data scientists to …

Reducing memory requirements of unsteady adjoint by synergistically using check‐pointing and compression

ASI Margetis, EM Papoutsis‐Kiachagias… - … Methods in Fluids, 2023 - Wiley Online Library
The unsteady adjoint method used in gradient‐based optimization in 2D and, particularly,
3D industrial problems modeled by unsteady PDEs may have significant storage …