A long time ago in the machine learning literature, the idea of incorporating a mechanism
inspired by the human visual system into neural networks was introduced. This idea is …

Despite their success, deep networks are used as black-box models with outputs that are not
easily explainable during the learning and the prediction phases. This lack of interpretability …

In the Transformer model," self-attention" combines information from attended embeddings
into the representation of the focal embedding in the next layer. Thus, across layers of the …

Attention mechanisms play a central role in NLP systems, especially within recurrent neural
network (RNN) models. Recently, there has been increasing interest in whether or not the …

In humans, Attention is a core property of all perceptual and cognitive operations. Given our
limited ability to process competing sources, attention mechanisms select, modulate, and …

The Transformer is a sequence model that forgoes traditional recurrent architectures in favor
of a fully attention-based approach. Besides improving performance, an advantage of using …

State-of-the-art natural language processing systems rely on supervision in the form of
annotated data to learn competent models. These models are generally trained on data in a …

Natural language understanding comprises a wide range of diverse tasks such as textual
entailment, question answering, semantic similarity assessment, and document …

Attention is an increasingly popular mechanism used in a wide range of neural
architectures. The mechanism itself has been realized in a variety of formats. However …

With the development of high computational devices, deep neural networks (DNNs), in
recent years, have gained significant popularity in many Artificial Intelligence (AI) …