Recent work has shown that deep reinforcement learning agents have difficulty in effectively
using their network parameters. We leverage prior insights into the advantages of sparse …

We explore the impact of parameter sparsity on the scaling behavior of Transformers trained
on massive datasets (ie," foundation models"), in both vision and language domains. In this …

Abstract In recent years, Dynamic Sparse Training (DST) has emerged as an alternative to
post-training pruning for generating efficient models. In principle, DST allows for a much …

In recent years, Dynamic Sparse Training (DST) has emerged as an alternative to post-
training pruning for generating efficient models. In principle, DST allows for a more memory …

This paper extends prior work on sparse tensor algebra compilers to generate
asymptotically efficient code for tensor expressions with affine subscript expressions. Our …

We present ELSA, a practical solution for creating deep networks that can easily be
deployed at different levels of sparsity. The core idea is to embed one or more sparse …