Tensor Attention, a multi-view attention that is able to capture high-order correlations among
multiple modalities, can overcome the representational limitations of classical matrix …

The self-attention mechanism is the key to the success of transformers in recent Large
Language Models (LLMs). However, the quadratic computational cost $ O (n^ 2) $ in the …

The computational complexity of the self-attention mechanism in popular transformer
architectures poses significant challenges for training and inference, and becomes the …

Large Language Models (LLMs) have demonstrated remarkable capabilities across various
applications, but their performance on long-context tasks is often limited by the …

In-context learning has been recognized as a key factor in the success of Large Language
Models (LLMs). It refers to the model's ability to learn patterns on the fly from provided in …

Large language models (LLMs) have had a profound impact on numerous aspects of daily
life including natural language processing, content generation, research methodologies and …

Recent empirical studies have identified fixed point iteration phenomena in deep neural
networks, where the hidden state tends to stabilize after several layers, showing minimal …

In this paper, we analyze the computational limitations of Mamba and State-space Models
(SSMs) by using the circuit complexity framework. Despite Mamba's stateful design and …

Modern Hopfield networks (MHNs) have emerged as powerful tools in deep learning,
capable of replacing components such as pooling layers, LSTMs, and attention …

Training neural networks is a fundamental problem in theoretical machine learning. Second-
order methods are rarely used in practice due to their high computational cost, even they …