No" zero-shot" without exponential data: Pretraining concept frequency determines multimodal model performance

V Udandarao, A Prabhu, A Ghosh… - The Thirty-eighth …, 2024 - openreview.net
Web-crawled pretraining datasets underlie the impressive" zero-shot" evaluation
performance of multimodal models, such as CLIP for classification and Stable-Diffusion for …

Clap4clip: Continual learning with probabilistic finetuning for vision-language models

S Jha, D Gong, L Yao - arxiv preprint arxiv:2403.19137, 2024 - arxiv.org
Continual learning (CL) aims to help deep neural networks learn new knowledge while
retaining what has been learned. Owing to their powerful generalizability, pre-trained vision …

A Practitioner's Guide to Continual Multimodal Pretraining

K Roth, V Udandarao, S Dziadzio, A Prabhu… - arxiv preprint arxiv …, 2024 - arxiv.org
Multimodal foundation models serve numerous applications at the intersection of vision and
language. Still, despite being pretrained on extensive data, they become outdated over time …

Just say the name: Online continual learning with category names only via data generation

M Seo, S Cho, M Lee, D Misra, H Choi, SJ Kim… - arxiv preprint arxiv …, 2024 - arxiv.org
Requiring extensive human supervision is often impractical for continual learning due to its
cost, leading to the emergence of'name-only continual learning'that only provides the name …

kNN-CLIP: Retrieval enables training-free segmentation on continually expanding large vocabularies

Z Gui, S Sun, R Li, J Yuan, Z An, K Roth… - arxiv preprint arxiv …, 2024 - arxiv.org
Continual segmentation has not yet tackled the challenge of improving open-vocabulary
segmentation models with training data for accurate segmentation across large, continually …

How to Merge Your Multimodal Models Over Time?

S Dziadzio, V Udandarao, K Roth, A Prabhu… - arxiv preprint arxiv …, 2024 - arxiv.org
Model merging combines multiple expert models-finetuned from a base foundation model
on diverse tasks and domains-into a single, more capable model. However, most existing …

Conformal-in-the-Loop for Learning with Imbalanced Noisy Data

JB Graham-Knight, J Fayyad, N Bayasi… - arxiv preprint arxiv …, 2024 - arxiv.org
Class imbalance and label noise are pervasive in large-scale datasets, yet much of machine
learning research assumes well-labeled, balanced data, which rarely reflects real world …