MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913557055012864 |
|---|---|
| author | Elbakary, Ahmed Issaid, Chaouki Ben ElBatt, Tamer Seddik, Karim Bennis, Mehdi |
| author_facet | Elbakary, Ahmed Issaid, Chaouki Ben ElBatt, Tamer Seddik, Karim Bennis, Mehdi |
| contents | In this paper, we introduce a method for fine-tuning Large Language Models (LLMs), inspired by Multi-Task learning in a federated manner. Our approach leverages the structure of each client's model and enables a learning scheme that considers other clients' tasks and data distribution. To mitigate the extensive computational and communication overhead often associated with LLMs, we utilize a parameter-efficient fine-tuning method, specifically Low-Rank Adaptation (LoRA), reducing the number of trainable parameters. Experimental results, with different datasets and models, demonstrate the proposed method's effectiveness compared to existing frameworks for federated fine-tuning of LLMs in terms of average and local performances. The proposed scheme outperforms existing baselines by achieving lower local loss for each client while maintaining comparable global performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_15524 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models Elbakary, Ahmed Issaid, Chaouki Ben ElBatt, Tamer Seddik, Karim Bennis, Mehdi Machine Learning Distributed, Parallel, and Cluster Computing In this paper, we introduce a method for fine-tuning Large Language Models (LLMs), inspired by Multi-Task learning in a federated manner. Our approach leverages the structure of each client's model and enables a learning scheme that considers other clients' tasks and data distribution. To mitigate the extensive computational and communication overhead often associated with LLMs, we utilize a parameter-efficient fine-tuning method, specifically Low-Rank Adaptation (LoRA), reducing the number of trainable parameters. Experimental results, with different datasets and models, demonstrate the proposed method's effectiveness compared to existing frameworks for federated fine-tuning of LLMs in terms of average and local performances. The proposed scheme outperforms existing baselines by achieving lower local loss for each client while maintaining comparable global performance. |
| title | MIRA: A Method of Federated MultI-Task Learning for LaRge LAnguage Models |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2410.15524 |