Transformer language models have received widespread public attention, yet their
generated text is often surprising even to NLP researchers. In this survey, we discuss over …

Transformers are arguably the main workhorse in recent natural language processing
research. By definition, a Transformer is invariant with respect to reordering of the input …

Abstract Knowledge-Enhanced Pre-trained Language Models (KEPLMs) are pre-trained
models with relation triples injecting from knowledge graphs to improve language …

We explore different ways to utilize position-based cross-attention in seq2seq networks to
enable length generalization in algorithmic tasks. We show that a simple approach of …

Recently, knowledge-enhanced pre-trained language models (KEPLMs) improve context-
aware representations via learning from structured relations in knowledge graphs, and/or …

Malware continues to evolve rapidly, and more than 450,000 new samples are captured
every day, which makes manual malware analysis impractical. However, existing deep …

Word order, an essential property of natural languages, is injected in Transformer-based
neural language models using position encoding. However, recent experiments have shown …

KEPLMs are pre-trained models that utilize external knowledge to enhance language
understanding. Previous language models facilitated knowledge acquisition by …

Position-based token-mixing approaches, such as FNet and MLPMixer, have shown to be
exciting attention alternatives for computer vision and natural language understanding. The …

The Transformer has emerged as the predominant model in Natural Language Processing
due to its exceptional performance in various sequence modeling tasks, particularly in …