Abstract Vision-and-language (VL) pre-training has proven to be highly effective on various
VL downstream tasks. While recent work has shown that fully transformer-based VL models …

Transformer is the state-of-the-art model in recent machine translation evaluations. Two
strands of research are promising to improve models of this kind: the first uses wide …

Transformer-based architectures have shown great success in image captioning, where
object regions are encoded and then attended into the vectorial representations to guide the …

Vision-Language (VL) models with the Two-Tower architecture have dominated visual-
language representation learning in recent years. Current VL models either use lightweight …

Self-attention networks have proven to be of profound value for its strength of capturing
global dependencies. In this work, we propose to model localness for self-attention …

Skip connection, is a widely-used technique to improve the performance and the
convergence of deep neural networks, which is believed to relieve the difficulty in …

Multi-head attention is appealing for the ability to jointly attend to information from different
representation subspaces at different positions. In this work, we introduce a disagreement …

Multi-head attention is appealing for the ability to jointly attend to information from different
representation subspaces at different positions. In this work, we propose two approaches to …

Self-attention networks (SANs) have drawn increasing interest due to their high
parallelization in computation and flexibility in modeling dependencies. SANs can be further …

Self-attention model has shown its flexibility in parallel computation and the effectiveness on
modeling both long-and short-term dependencies. However, it calculates the dependencies …