We address the problem of retrieving in-the-wild images with both a sketch and a text query.
We present TASK-former (Text And SKetch transformer), an end-to-end trainable model for …

In the real world, where information is abundant and diverse across different modalities,
understanding and utilizing various data types to improve retrieval systems is a key focus of …

Jointly using text and sketch for scene-level image retrieval utilizes the complementary
between text and sketch to describe the fine-grained scene content and retrieve the target …

In this paper, we build on research which has applied visually-derived features for grounded
semantics by leveraging an additional modality: simulated hand muscle activations. We …

Previous work on multimodality in interaction has mostly focussed on integrating models for
verbal utterances and embodied modalities like gestures. In this paper, we take a first step …

Sketch Based Image Retrieval (SBIR) is a challenging problem mainly due to a significant
cross-domain gap between hand-drawn sketches and natural images. While extra semantic …

Socially competent robots should be equipped with the ability to perceive the world that
surrounds them and communicate about it in a human-like manner. Representative skills …

A lot of recent work in Language & Vision has looked at generating descriptions or referring
expressions for objects in scenes of real-world images, though focusing mostly on relatively …