Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.19534 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916864736624640 |
|---|---|
| author | Shakeri, Ali Zhang, Wei Emma Beheshti, Amin Chen, Weitong Yang, Jian Yang, Lishan |
| author_facet | Shakeri, Ali Zhang, Wei Emma Beheshti, Amin Chen, Weitong Yang, Jian Yang, Lishan |
| contents | Pre-trained Language Models (PLMs) have demonstrated impressive performance in various NLP tasks. However, traditional fine-tuning methods for leveraging PLMs for downstream tasks entail significant computational overhead. Prompt-tuning has emerged as an efficient alternative that involves prepending a limited number of parameters to the input sequence and only updating them while the PLM's parameters are frozen. However, this technique's prompts remain fixed for all inputs, reducing the model's flexibility. The Federated Learning (FL) technique has gained attention in recent years to address the growing concerns around data privacy. However, challenges such as communication and computation limitations of clients still need to be addressed. To mitigate these challenges, this paper introduces the Federated Dynamic Prompt Generator (FedDPG), which incorporates a dynamic prompt generator network to generate context-aware prompts based on the given input, ensuring flexibility and adaptability while prioritising data privacy in federated learning settings. Our experiments on three NLP benchmark datasets showcase that FedDPG outperforms the state-of-the-art parameter-efficient fine-tuning methods in terms of global model performance, and has significantly reduced the calculation time and the number of parameters to be sent through the FL network. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_19534 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FedDPG: An Adaptive Yet Efficient Prompt-tuning Approach in Federated Learning Settings Shakeri, Ali Zhang, Wei Emma Beheshti, Amin Chen, Weitong Yang, Jian Yang, Lishan Machine Learning Artificial Intelligence Computation and Language I.2; I.7 Pre-trained Language Models (PLMs) have demonstrated impressive performance in various NLP tasks. However, traditional fine-tuning methods for leveraging PLMs for downstream tasks entail significant computational overhead. Prompt-tuning has emerged as an efficient alternative that involves prepending a limited number of parameters to the input sequence and only updating them while the PLM's parameters are frozen. However, this technique's prompts remain fixed for all inputs, reducing the model's flexibility. The Federated Learning (FL) technique has gained attention in recent years to address the growing concerns around data privacy. However, challenges such as communication and computation limitations of clients still need to be addressed. To mitigate these challenges, this paper introduces the Federated Dynamic Prompt Generator (FedDPG), which incorporates a dynamic prompt generator network to generate context-aware prompts based on the given input, ensuring flexibility and adaptability while prioritising data privacy in federated learning settings. Our experiments on three NLP benchmark datasets showcase that FedDPG outperforms the state-of-the-art parameter-efficient fine-tuning methods in terms of global model performance, and has significantly reduced the calculation time and the number of parameters to be sent through the FL network. |
| title | FedDPG: An Adaptive Yet Efficient Prompt-tuning Approach in Federated Learning Settings |
| topic | Machine Learning Artificial Intelligence Computation and Language I.2; I.7 |
| url | https://arxiv.org/abs/2507.19534 |