Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.11793 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916696487362560 |
|---|---|
| author | Li, Yue Zhang, Lihong |
| author_facet | Li, Yue Zhang, Lihong |
| contents | Federated Learning (FL) faces major challenges regarding communication overhead and model privacy when training large language models (LLMs), especially in healthcare applications. To address these, we introduce Selective Attention Federated Learning (SAFL), a novel approach that dynamically fine-tunes only those transformer layers identified as attention-critical. By employing attention patterns to determine layer importance, SAFL significantly reduces communication bandwidth and enhances differential privacy resilience. Evaluations on clinical NLP benchmarks (i2b2 Clinical Concept Extraction and MIMIC-III discharge summaries) demonstrate that SAFL achieves competitive performance with centralized models while substantially improving communication efficiency and privacy preservation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_11793 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Selective Attention Federated Learning: Improving Privacy and Efficiency for Clinical Text Classification Li, Yue Zhang, Lihong Computation and Language Artificial Intelligence Federated Learning (FL) faces major challenges regarding communication overhead and model privacy when training large language models (LLMs), especially in healthcare applications. To address these, we introduce Selective Attention Federated Learning (SAFL), a novel approach that dynamically fine-tunes only those transformer layers identified as attention-critical. By employing attention patterns to determine layer importance, SAFL significantly reduces communication bandwidth and enhances differential privacy resilience. Evaluations on clinical NLP benchmarks (i2b2 Clinical Concept Extraction and MIMIC-III discharge summaries) demonstrate that SAFL achieves competitive performance with centralized models while substantially improving communication efficiency and privacy preservation. |
| title | Selective Attention Federated Learning: Improving Privacy and Efficiency for Clinical Text Classification |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.11793 |