Continual learning of large language models: A comprehensive survey

H Shi, Z Xu, H Wang, W Qin, W Wang, Y Wang… - arxiv preprint arxiv …, 2024 - arxiv.org
The recent success of large language models (LLMs) trained on static, pre-collected,
general datasets has sparked numerous research directions and applications. One such …

Recent advances of foundation language models-based continual learning: A survey

Y Yang, J Zhou, X Ding, T Huai, S Liu, Q Chen… - ACM Computing …, 2024 - dl.acm.org
Recently, foundation language models (LMs) have marked significant achievements in the
domains of natural language processing (NLP) and computer vision (CV). Unlike traditional …

A survey on model moerging: Recycling and routing among specialized experts for collaborative learning

P Yadav, C Raffel, M Muqeeth, L Caccia, H Liu… - arxiv preprint arxiv …, 2024 - arxiv.org
The availability of performant pre-trained models has led to a proliferation of fine-tuned
expert models that are specialized to a particular domain or task. Model MoErging methods …

Select and distill: Selective dual-teacher knowledge transfer for continual learning on vision-language models

YC Yu, CP Huang, JJ Chen, KP Chang, YH Lai… - … on Computer Vision, 2024 - Springer
Large-scale vision-language models (VLMs) have shown a strong zero-shot generalization
capability on unseen-domain data. However, adapting pre-trained VLMs to a sequence of …

A Practitioner's Guide to Continual Multimodal Pretraining

K Roth, V Udandarao, S Dziadzio, A Prabhu… - arxiv preprint arxiv …, 2024 - arxiv.org
Multimodal foundation models serve numerous applications at the intersection of vision and
language. Still, despite being pretrained on extensive data, they become outdated over time …

Clip with generative latent replay: a strong baseline for incremental learning

E Frascaroli, A Panariello, P Buzzega… - arxiv preprint arxiv …, 2024 - arxiv.org
With the emergence of Transformers and Vision-Language Models (VLMs) such as CLIP,
fine-tuning large pre-trained models has recently become a prevalent strategy in Continual …

Theory on mixture-of-experts in continual learning

H Li, S Lin, L Duan, Y Liang, NB Shroff - arxiv preprint arxiv:2406.16437, 2024 - arxiv.org
Continual learning (CL) has garnered significant attention because of its ability to adapt to
new tasks that arrive over time. Catastrophic forgetting (of old tasks) has been identified as a …

Llms can evolve continually on modality for x-modal reasoning

J Yu, H **ong, L Zhang, H Diao, Y Zhuge… - arxiv preprint arxiv …, 2024 - arxiv.org
Multimodal Large Language Models (MLLMs) have gained significant attention due to their
impressive capabilities in multimodal understanding. However, existing methods rely heavily …

Advancing Cross-domain Discriminability in Continual Learning of Vision-Language Models

Y Xu, Y Chen, J Nie, Y Wang, H Zhuang… - arxiv preprint arxiv …, 2024 - arxiv.org
Continual learning (CL) with Vision-Language Models (VLMs) has overcome the constraints
of traditional CL, which only focuses on previously encountered classes. During the CL of …

Trends and challenges of real-time learning in large language models: A critical review

M Jovanovic, P Voss - arxiv preprint arxiv:2404.18311, 2024 - arxiv.org
Real-time learning concerns the ability of learning systems to acquire knowledge over time,
enabling their adaptation and generalization to novel tasks. It is a critical ability for …