While pretraining on large-scale image-text data from the Web has facilitated rapid progress
on many vision-and-language (V&L) tasks, recent work has demonstrated that pretrained …

Abstract This paper introduces Grounded Image Text Matching with Mismatched Relation
(GITM-MR), a novel visual-linguistic joint task that evaluates the relation understanding …

Pre-trained vision-language models (VLMs) have achieved high performance on various
downstream tasks which have been widely used for visual grounding tasks in a weakly …

Vision-language foundation models have had considerable increase in performances in the
last few years. However, there is still a lack of comprehensive evaluation methods able to …

Recent work in vision-and-language pretraining has investigated supervised signals from
object detection data to learn better, fine-grained multimodal representations. In this work …

We present an approach to evaluate the robustness of pre-trained vision and language
(V&L) models to noise in input data. Given a source image/text, we perturb it using standard …

What is sentence meaning and its ideal representation? Much of the expressive power of
human language derives from semantic composition, the mind's ability to represent meaning …

Pretrained visual-language models have made significant advancements in multimodal
tasks, including image-text retrieval. However, a major challenge in image-text matching lies …

Recent vision-language foundation models still frequently produce outputs misaligned with
their inputs, evidenced by object hallucination in captioning and prompt misalignment in the …

Visual reasoning requires models to construct a reasoning process towards the final
decision. Previous studies have used attention maps or textual explanations to illustrate the …