The field of natural language processing has seen impressive progress in recent years, with
neural network models replacing many of the traditional systems. A plethora of new models …

Transformers are emerging as the new workhorse of NLP, showing great success across
tasks. Unlike LSTMs, transformers process input sequences entirely through self-attention …

What is the computational model behind a Transformer? Where recurrent neural networks
have direct parallels in finite state machines, allowing clear discussion and thought around …

Despite their impressive performance in NLP, self-attention networks were recently proved
to be limited for processing formal languages with hierarchical structure, such as $\mathsf …

Despite the success of language models using neural networks, it remains unclear to what
extent neural models have the generalization ability to perform inferences. In this paper, we …

We focus on the recognition of Dyck-n ($\mathcal {D} _n $) languages with self-attention
(SA) networks, which has been deemed to be a difficult task for these networks. We compare …

While long short-term memory (LSTM) neural net architectures are designed to capture
sequence information, human language is generally composed of hierarchical structures …

We introduce three memory-augmented Recurrent Neural Networks (MARNNs) and explore
their capabilities on a series of simple language modeling tasks whose solutions require …

In recent years, the discussion about systematicity of neural network learning has gained
renewed interest, in particular the formal analysis of neural network behaviour. In this paper …

The generalized Dyck language has been used to analyze the ability of Recurrent Neural
Networks (RNNs) to learn context-free grammars (CFGs). Recent studies draw conflicting …