FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909929784213504 |
|---|---|
| author | Gao, Yan Scamarcia, Massimo Roberto Fernandez-Marques, Javier Naseri, Mohammad Ng, Chong Shen Stripelis, Dimitris Li, Zexi Shen, Tao Bai, Jiamu Chen, Daoyuan Zhang, Zikai Hu, Rui Song, InSeo KangYoon, Lee Jia, Hong Dang, Ting Wang, Junyan Liu, Zheyuan Beutel, Daniel Janes Lyu, Lingjuan Lane, Nicholas D. |
| author_facet | Gao, Yan Scamarcia, Massimo Roberto Fernandez-Marques, Javier Naseri, Mohammad Ng, Chong Shen Stripelis, Dimitris Li, Zexi Shen, Tao Bai, Jiamu Chen, Daoyuan Zhang, Zikai Hu, Rui Song, InSeo KangYoon, Lee Jia, Hong Dang, Ting Wang, Junyan Liu, Zheyuan Beutel, Daniel Janes Lyu, Lingjuan Lane, Nicholas D. |
| contents | Large Language Models (LLMs) have achieved state-of-the-art results across diverse domains, yet their development remains reliant on vast amounts of publicly available data, raising concerns about data scarcity and the lack of access to domain-specific, sensitive information. Federated Learning (FL) presents a compelling framework to address these challenges by enabling decentralized fine-tuning on pre-trained LLMs without sharing raw data. However, the compatibility and performance of pre-trained LLMs in FL settings remain largely under explored. We introduce the FlowerTune LLM Leaderboard, a first-of-its-kind benchmarking suite designed to evaluate federated fine-tuning of LLMs across four diverse domains: general NLP, finance, medical, and coding. Each domain includes federated instruction-tuning datasets and domain-specific evaluation metrics. Our results, obtained through a collaborative, open-source and community-driven approach, provide the first comprehensive comparison across 26 pre-trained LLMs with different aggregation and fine-tuning strategies under federated settings, offering actionable insights into model performance, resource constraints, and domain adaptation. This work lays the foundation for developing privacy-preserving, domain-specialized LLMs for real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02961 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models Gao, Yan Scamarcia, Massimo Roberto Fernandez-Marques, Javier Naseri, Mohammad Ng, Chong Shen Stripelis, Dimitris Li, Zexi Shen, Tao Bai, Jiamu Chen, Daoyuan Zhang, Zikai Hu, Rui Song, InSeo KangYoon, Lee Jia, Hong Dang, Ting Wang, Junyan Liu, Zheyuan Beutel, Daniel Janes Lyu, Lingjuan Lane, Nicholas D. Computation and Language Large Language Models (LLMs) have achieved state-of-the-art results across diverse domains, yet their development remains reliant on vast amounts of publicly available data, raising concerns about data scarcity and the lack of access to domain-specific, sensitive information. Federated Learning (FL) presents a compelling framework to address these challenges by enabling decentralized fine-tuning on pre-trained LLMs without sharing raw data. However, the compatibility and performance of pre-trained LLMs in FL settings remain largely under explored. We introduce the FlowerTune LLM Leaderboard, a first-of-its-kind benchmarking suite designed to evaluate federated fine-tuning of LLMs across four diverse domains: general NLP, finance, medical, and coding. Each domain includes federated instruction-tuning datasets and domain-specific evaluation metrics. Our results, obtained through a collaborative, open-source and community-driven approach, provide the first comprehensive comparison across 26 pre-trained LLMs with different aggregation and fine-tuning strategies under federated settings, offering actionable insights into model performance, resource constraints, and domain adaptation. This work lays the foundation for developing privacy-preserving, domain-specialized LLMs for real-world applications. |
| title | FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.02961 |