FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910810195886080 |
|---|---|
| author | Jiang, Wenzheng Wang, Ji Zhang, Xiongtao Bao, Weidong Tan, Cheston Fan, Flint Xiaofeng |
| author_facet | Jiang, Wenzheng Wang, Ji Zhang, Xiongtao Bao, Weidong Tan, Cheston Fan, Flint Xiaofeng |
| contents | Federated Reinforcement Learning (FedRL) improves sample efficiency while preserving privacy; however, most existing studies assume homogeneous agents, limiting its applicability in real-world scenarios. This paper investigates FedRL in black-box settings with heterogeneous agents, where each agent employs distinct policy networks and training configurations without disclosing their internal details. Knowledge Distillation (KD) is a promising method for facilitating knowledge sharing among heterogeneous models, but it faces challenges related to the scarcity of public datasets and limitations in knowledge representation when applied to FedRL. To address these challenges, we propose Federated Heterogeneous Policy Distillation (FedHPD), which solves the problem of heterogeneous FedRL by utilizing action probability distributions as a medium for knowledge sharing. We provide a theoretical analysis of FedHPD's convergence under standard assumptions. Extensive experiments corroborate that FedHPD shows significant improvements across various reinforcement learning benchmark tasks, further validating our theoretical findings. Moreover, additional experiments demonstrate that FedHPD operates effectively without the need for an elaborate selection of public datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00870 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation Jiang, Wenzheng Wang, Ji Zhang, Xiongtao Bao, Weidong Tan, Cheston Fan, Flint Xiaofeng Machine Learning Artificial Intelligence Multiagent Systems I.2.11 Federated Reinforcement Learning (FedRL) improves sample efficiency while preserving privacy; however, most existing studies assume homogeneous agents, limiting its applicability in real-world scenarios. This paper investigates FedRL in black-box settings with heterogeneous agents, where each agent employs distinct policy networks and training configurations without disclosing their internal details. Knowledge Distillation (KD) is a promising method for facilitating knowledge sharing among heterogeneous models, but it faces challenges related to the scarcity of public datasets and limitations in knowledge representation when applied to FedRL. To address these challenges, we propose Federated Heterogeneous Policy Distillation (FedHPD), which solves the problem of heterogeneous FedRL by utilizing action probability distributions as a medium for knowledge sharing. We provide a theoretical analysis of FedHPD's convergence under standard assumptions. Extensive experiments corroborate that FedHPD shows significant improvements across various reinforcement learning benchmark tasks, further validating our theoretical findings. Moreover, additional experiments demonstrate that FedHPD operates effectively without the need for an elaborate selection of public datasets. |
| title | FedHPD: Heterogeneous Federated Reinforcement Learning via Policy Distillation |
| topic | Machine Learning Artificial Intelligence Multiagent Systems I.2.11 |
| url | https://arxiv.org/abs/2502.00870 |