FedHFT: Efficient Federated Finetuning with Heterogeneous Edge Clients
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915556948443136 |
|---|---|
| author | Ilhan, Fatih Tekin, Selim Furkan Huang, Tiansheng Liu, Gaowen Kompella, Ramana Eisenhauer, Greg Lin, Yingyan Celine Pu, Calton Liu, Ling |
| author_facet | Ilhan, Fatih Tekin, Selim Furkan Huang, Tiansheng Liu, Gaowen Kompella, Ramana Eisenhauer, Greg Lin, Yingyan Celine Pu, Calton Liu, Ling |
| contents | Fine-tuning pre-trained large language models (LLMs) has become a common practice for personalized natural language understanding (NLU) applications on downstream tasks and domain-specific datasets. However, there are two main challenges: (i) limited and/or heterogeneous data for fine-tuning due to proprietary data confidentiality or privacy requirements, and (ii) varying computation resources available across participating clients such as edge devices. This paper presents FedHFT - an efficient and personalized federated fine-tuning framework to address both challenges. First, we introduce a mixture of masked adapters to handle resource heterogeneity across participating clients, enabling high-performance collaborative fine-tuning of pre-trained language model(s) across multiple clients in a distributed setting, while keeping proprietary data local. Second, we introduce a bi-level optimization approach to handle non-iid data distribution based on masked personalization and client clustering. Extensive experiments demonstrate significant performance and efficiency improvements over various natural language understanding tasks under data and resource heterogeneity compared to representative heterogeneous federated learning methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_14054 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FedHFT: Efficient Federated Finetuning with Heterogeneous Edge Clients Ilhan, Fatih Tekin, Selim Furkan Huang, Tiansheng Liu, Gaowen Kompella, Ramana Eisenhauer, Greg Lin, Yingyan Celine Pu, Calton Liu, Ling Machine Learning Distributed, Parallel, and Cluster Computing Fine-tuning pre-trained large language models (LLMs) has become a common practice for personalized natural language understanding (NLU) applications on downstream tasks and domain-specific datasets. However, there are two main challenges: (i) limited and/or heterogeneous data for fine-tuning due to proprietary data confidentiality or privacy requirements, and (ii) varying computation resources available across participating clients such as edge devices. This paper presents FedHFT - an efficient and personalized federated fine-tuning framework to address both challenges. First, we introduce a mixture of masked adapters to handle resource heterogeneity across participating clients, enabling high-performance collaborative fine-tuning of pre-trained language model(s) across multiple clients in a distributed setting, while keeping proprietary data local. Second, we introduce a bi-level optimization approach to handle non-iid data distribution based on masked personalization and client clustering. Extensive experiments demonstrate significant performance and efficiency improvements over various natural language understanding tasks under data and resource heterogeneity compared to representative heterogeneous federated learning methods. |
| title | FedHFT: Efficient Federated Finetuning with Heterogeneous Edge Clients |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2510.14054 |