Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910014508105728 |
|---|---|
| author | Phung, Thu Hang Nguyen, Duong M. Huynh, Thanh Trung Nguyen, Quoc Viet Hung Hoang, Trong Nghia Nguyen, Phi Le |
| author_facet | Phung, Thu Hang Nguyen, Duong M. Huynh, Thanh Trung Nguyen, Quoc Viet Hung Hoang, Trong Nghia Nguyen, Phi Le |
| contents | This paper introduces a generalized federated prompt-tuning framework for practical scenarios where local datasets are multi-modal and exhibit different distributional patterns of missing features at the input level. The proposed framework bridges the gap between federated learning and multi-modal prompt-tuning which have traditionally focused on either uni-modal or centralized data. A key challenge in this setting arises from the lack of semantic alignment between prompt instructions that encode similar distributional patterns of missing data across different clients. To address this, our framework introduces specialized client-tuning and server-aggregation designs that simultaneously optimize, align, and aggregate prompt-tuning instructions across clients and data modalities. This allows prompt instructions to complement one another and be combined effectively. Extensive evaluations on diverse multimodal benchmark datasets demonstrate that our work consistently outperforms state-of-the-art (SOTA) baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_07081 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data Phung, Thu Hang Nguyen, Duong M. Huynh, Thanh Trung Nguyen, Quoc Viet Hung Hoang, Trong Nghia Nguyen, Phi Le Multimedia Artificial Intelligence Computer Vision and Pattern Recognition This paper introduces a generalized federated prompt-tuning framework for practical scenarios where local datasets are multi-modal and exhibit different distributional patterns of missing features at the input level. The proposed framework bridges the gap between federated learning and multi-modal prompt-tuning which have traditionally focused on either uni-modal or centralized data. A key challenge in this setting arises from the lack of semantic alignment between prompt instructions that encode similar distributional patterns of missing data across different clients. To address this, our framework introduces specialized client-tuning and server-aggregation designs that simultaneously optimize, align, and aggregate prompt-tuning instructions across clients and data modalities. This allows prompt instructions to complement one another and be combined effectively. Extensive evaluations on diverse multimodal benchmark datasets demonstrate that our work consistently outperforms state-of-the-art (SOTA) baselines. |
| title | Federated Prompt-Tuning with Heterogeneous and Incomplete Multimodal Client Data |
| topic | Multimedia Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2602.07081 |