Outlier Features (OFs) are neurons whose activation magnitudes significantly exceed the
average over a neural network's (NN) width. They are well known to emerge during standard …

AdamW has been the default optimizer for transformer pretraining. For many years, our
community searches for faster and more stable optimizers with only constraint positive …

We present OLMo 2, the next generation of our fully open language models. OLMo 2
includes dense autoregressive models with improved architecture and training recipe …

We introduce\textbf {Gr} adient Descent with\textbf {A} daptive\textbf {M} omentum\textbf {S}
caling (\textbf {Grams}), a novel optimization algorithm that decouples the direction and …

Multi-task learning through composite loss functions is fundamental to modern deep
learning, yet optimizing competing objectives remains challenging. We present new …

Curvature regularization techniques like Sharpness Aware Minimization (SAM) have shown
great promise in improving generalization on vision tasks. However, we find that SAM …

This paper introduces Moonshine, a family of speech recognition models optimized for live
transcription and voice command processing. Moonshine is based on an encoder-decoder …

We aim to understand physics of skill learning, ie, how skills are learned in neural networks
during training. We start by observing the Domino effect, ie, skills are learned sequentially …

Layer-wise preconditioning methods are a family of memory-efficient optimization algorithms
that introduce preconditioners per axis of each layer's weight tensors. These methods have …

Modern adaptive optimization methods, such as Adam and its variants, have emerged as the
most widely used tools in deep learning over recent years. These algorithms offer automatic …