Movable Antenna Enhanced Federated Fine-Tuning of Large Language Models via Hybrid Client Selection Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912671408848896 |
|---|---|
| author | Zhao, Yang Xiu, Yue Dai, Chengxiao Wei, Ning Niyato, Dusit |
| author_facet | Zhao, Yang Xiu, Yue Dai, Chengxiao Wei, Ning Niyato, Dusit |
| contents | Federated fine-tuning of large language models (LLMs) over bandwidth-limited 6G links must meet strict round-time and energy budgets. Analog over-the-air (OTA) aggregation reduces uplink cost but is sensitive to fading and interference, which distort the aggregated gradient. We consider a two-phase workflow (centralized pre-training followed by federated fine-tuning) where the base station uses a movable-antenna (MA) array. In each round, MA element positions and the receive/transmit beamformers are adjusted under minimum-spacing constraints to reshape the channel and improve OTA aggregation without increasing user transmit power. We formulate a mixed-integer, nonconvex resource-allocation problem that jointly selects clients and optimizes the number of global rounds, CPU frequencies, mini-batch sizes, MA positions, and analog weights under end-to-end latency and energy limits. A successive convex approximation-penalty dual decomposition (SCA-PDD) routine alternates convex updates with oblique-manifold beamforming and spacing-aware MA placement. Experiments on OpenLLaMA-v2 (3B) with LoRA and 4-bit quantization on Alpaca and Dolly (10 clients) attain round-30 validation perplexities as low as 2.94 (Alpaca, K=1) and 4.62 (Dolly, K=1). Relative to the strongest non-MA baseline at the same concurrency, this corresponds to 17.4 percent (Alpaca, K=1) and 54.4 percent (Dolly, K=1) lower perplexity; at K=2 the reductions are 14.2 percent (Alpaca) and 13.7 percent (Dolly). Participation fairness also improves across all uplink concurrencies K in {1,2,4,8}, with the largest margins when fewer clients transmit per round. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_00011 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Movable Antenna Enhanced Federated Fine-Tuning of Large Language Models via Hybrid Client Selection Optimization Zhao, Yang Xiu, Yue Dai, Chengxiao Wei, Ning Niyato, Dusit Signal Processing Federated fine-tuning of large language models (LLMs) over bandwidth-limited 6G links must meet strict round-time and energy budgets. Analog over-the-air (OTA) aggregation reduces uplink cost but is sensitive to fading and interference, which distort the aggregated gradient. We consider a two-phase workflow (centralized pre-training followed by federated fine-tuning) where the base station uses a movable-antenna (MA) array. In each round, MA element positions and the receive/transmit beamformers are adjusted under minimum-spacing constraints to reshape the channel and improve OTA aggregation without increasing user transmit power. We formulate a mixed-integer, nonconvex resource-allocation problem that jointly selects clients and optimizes the number of global rounds, CPU frequencies, mini-batch sizes, MA positions, and analog weights under end-to-end latency and energy limits. A successive convex approximation-penalty dual decomposition (SCA-PDD) routine alternates convex updates with oblique-manifold beamforming and spacing-aware MA placement. Experiments on OpenLLaMA-v2 (3B) with LoRA and 4-bit quantization on Alpaca and Dolly (10 clients) attain round-30 validation perplexities as low as 2.94 (Alpaca, K=1) and 4.62 (Dolly, K=1). Relative to the strongest non-MA baseline at the same concurrency, this corresponds to 17.4 percent (Alpaca, K=1) and 54.4 percent (Dolly, K=1) lower perplexity; at K=2 the reductions are 14.2 percent (Alpaca) and 13.7 percent (Dolly). Participation fairness also improves across all uplink concurrencies K in {1,2,4,8}, with the largest margins when fewer clients transmit per round. |
| title | Movable Antenna Enhanced Federated Fine-Tuning of Large Language Models via Hybrid Client Selection Optimization |
| topic | Signal Processing |
| url | https://arxiv.org/abs/2506.00011 |