Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning

W Kuang, B Qian, Z Li, D Chen, D Gao, X Pan… - Proceedings of the 30th …, 2024 - dl.acm.org
Large language models (LLMs) have demonstrated great capabilities in various natural
language understanding and generation tasks. These pre-trained LLMs can be further …

Data-juicer: A one-stop data processing system for large language models

D Chen, Y Huang, Z Ma, H Chen, X Pan, C Ge… - Companion of the 2024 …, 2024 - dl.acm.org
The immense evolution in Large Language Models (LLMs) has underscored the importance
of massive, heterogeneous, and high-quality data. A data recipe is a mixture of data from …

Decentralized federated learning: A survey and perspective

L Yuan, Z Wang, L Sun, SY Philip… - IEEE Internet of Things …, 2024 - ieeexplore.ieee.org
Federated learning (FL) has been gaining attention for its ability to share knowledge while
maintaining user data, protecting privacy, increasing learning efficiency, and reducing …

Federated large language models: Current progress and future directions

Y Yao, J Zhang, J Wu, C Huang, Y **a, T Yu… - arxiv preprint arxiv …, 2024 - arxiv.org
Large language models are rapidly gaining popularity and have been widely adopted in real-
world applications. While the quality of training data is essential, privacy concerns arise …

A survey of resource-efficient llm and multimodal foundation models

M Xu, W Yin, D Cai, R Yi, D Xu, Q Wang, B Wu… - arxiv preprint arxiv …, 2024 - arxiv.org
Large foundation models, including large language models (LLMs), vision transformers
(ViTs), diffusion, and LLM-based multimodal models, are revolutionizing the entire machine …

On the convergence of zeroth-order federated tuning for large language models

Z Ling, D Chen, L Yao, Y Li, Y Shen - Proceedings of the 30th ACM …, 2024 - dl.acm.org
The confluence of Federated Learning (FL) and Large Language Models (LLMs) is ushering
in a new era in privacy-preserving natural language processing. However, the intensive …

Towards Federated Large Language Models: Motivations, Methods, and Future Directions

Y Cheng, W Zhang, Z Zhang, C Zhang… - … Surveys & Tutorials, 2024 - ieeexplore.ieee.org
Large Language Models (LLMs), such as LLaMA and GPT-4, have transformed the
paradigm of natural language comprehension and generation. Despite their impressive …

Fwdllm: Efficient federated finetuning of large language models with perturbed inferences

M Xu, D Cai, Y Wu, X Li, S Wang - … of the 2024 USENIX Conference on …, 2024 - dl.acm.org
Large Language Models (LLMs) are transforming the landscape of mobile intelligence.
Federated Learning (FL), a method to preserve user data privacy, is often employed in fine …

Federated fine-tuning of large language models under heterogeneous language tasks and client resources

J Bai, D Chen, B Qian, L Yao, Y Li - arxiv e-prints, 2024 - ui.adsabs.harvard.edu
Federated Learning (FL) has recently been applied to the parameter-efficient fine-tuning of
Large Language Models (LLMs). While promising, it raises significant challenges due to the …

On the limitations of compute thresholds as a governance strategy

S Hooker - arxiv preprint arxiv:2407.05694, 2024 - arxiv.org
At face value, this essay is about understanding a fairly esoteric governance tool called
compute thresholds. However, in order to grapple with whether these thresholds will achieve …