The development of video large multimodal models (LMMs) has been hindered by the
difficulty of curating large amounts of high-quality raw data from the web. To address this, we …

Long-context capability is critical for multi-modal foundation models, especially for long
video understanding. We introduce LongVILA, a full-stack solution for long-context visual …

Abstract Empowered by Large Language Models (LLMs), recent advancements in Video-
based LLMs (VideoLLMs) have driven progress in various video understanding tasks. These …

Self-attention mechanism is the key of the Transformer but often criticized for its computation
demands. Previous token pruning works motivate their methods from the view of …

Video summarization aims to create short, accurate, and cohesive summaries of longer
videos. Despite the existence of various video summarization datasets, a notable limitation …

Contrastive Language-Image Pre-training (CLIP) has been widely studied and applied in
numerous applications. However, the emphasis on brief summary texts during pre-training …

Audio descriptions (ADs) function as acoustic commentaries designed to assist blind
persons and persons with visual impairments in accessing digital media content on …

Transforming recorded videos into concise and accurate textual summaries is a growing
challenge in multimodal learning. This paper introduces VISTA, a dataset specifically …

This paper introduces the COCONut-PanCap dataset, created to enhance panoptic
segmentation and grounded image captioning. Building upon the COCO dataset with …

Video is an increasingly prominent and information-dense medium, yet it poses substantial
challenges for language models. A typical video consists of a sequence of shorter segments …