Models using structured state space sequence (S4) layers have achieved state-of-the-art
performance on long-range sequence modeling tasks. An S4 layer combines linear state …

Abstract Machine learning models are increasingly being scaled in both sequence length
and model dimension to reach longer contexts and better performance. However, existing …

The design choices in the Transformer attention mechanism, including weak inductive bias
and quadratic computational complexity, have limited its application for modeling long …

Transformers have quickly shined in the computer vision world since the emergence of
Vision Transformers (ViTs). The dominant role of convolutional neural networks (CNNs) …

Partial differential equations (PDEs) are central to describing complex physical system
simulations. Their expensive solution techniques have led to an increased interest in deep …

In computer vision, 2D convolution is arguably the most important operation performed by a
ConvNet. Unsurprisingly, it has been the focus of intense software and hardware …

Spiking neural networks (SNNs) take inspiration from the brain to enable energy-efficient
computations. Since the advent of Transformers, SNNs have struggled to compete with …

Mutations that affect RNA splicing significantly impact human diversity and disease. Here we
present a method using transformers, a type of machine learning model, to detect splicing …

We present a new convolution layer for deep learning architectures which we call
QuadConv—an approximation to continuous convolution via quadrature. Our operator is …

We present Differentiable Neural Architectures (DNArch), a method that jointly learns the
weights and the architecture of Convolutional Neural Networks (CNNs) by backpropagation …