Abstract We present Ego-Exo4D a diverse large-scale multimodal multiview video dataset
and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric …

Abstract Large Language Models (LLMs) have been enhanced with vision capabilities
enabling them to comprehend images videos and interleaved vision-language content …

Surgical video-language pretraining (VLP) faces unique challenges due to the knowledge
domain gap and the scarcity of multi-modal data. This study aims to bridge the gap by …

The egocentric and exocentric viewpoints of a human activity look dramatically different, yet
invariant representations to link them are essential for many potential applications in …

Procedural activity understanding requires perceiving human actions in terms of a broader
task, where multiple keysteps are performed in sequence across a long video to reach a …

We address the problem of online action segmentation for egocentric procedural task
videos. While previous studies have mostly focused on offline action segmentation where …

We introduce HT-Step, a large-scale dataset containing temporal annotations of instructional
article steps in cooking videos. It includes 122k segment-level annotations over 20k narrated …

A well-known dilemma in large vision-language models (eg, GPT-4, LLaVA) is that while
increasing the number of vision tokens generally enhances visual understanding, it also …

In this paper we present an approach for localizing steps of procedural activities in narrated
how-to videos. To deal with the scarcity of labeled data at scale, we source the step …

Voice assistants capable of answering user queries during various physical tasks have
shown promise in guiding users through complex procedures. However, users often find it …