Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.19864 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917939076136960 |
|---|---|
| author | Li, Liang Chen, Xiaopei Wu, Wen |
| author_facet | Li, Liang Chen, Xiaopei Wu, Wen |
| contents | To enable large model (LM) based edge intelligent service provisioning, on-device fine-tuning with locally personalized data allows for continuous and privacy-preserving LM customization. In this paper, we propose RingAda, a collaborative training framework designed for fine-tuning transformer-based LMs on edge devices. Particularly, RingAda performs parameter-efficient adapter fine-tuning across a set of interconnected edge devices, forming a ring topology for per-batch training by sequentially placing frozen transformer blocks and their trainable adapter modules on the devices. RingAda follows a novel pipeline-parallel training mechanism with top-down adapter unfreezing, allowing for early-stopping of backpropagation at the lowest unfrozen adapter layer, thereby accelerating the fine-tuning process. Extensive experimental results demonstrate that RingAda significantly reduces fine-tuning time and memory costs while maintaining competitive model performance compared to its peer designs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19864 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RingAda: Pipelining Large Model Fine-Tuning on Edge Devices with Scheduled Layer Unfreezing Li, Liang Chen, Xiaopei Wu, Wen Distributed, Parallel, and Cluster Computing To enable large model (LM) based edge intelligent service provisioning, on-device fine-tuning with locally personalized data allows for continuous and privacy-preserving LM customization. In this paper, we propose RingAda, a collaborative training framework designed for fine-tuning transformer-based LMs on edge devices. Particularly, RingAda performs parameter-efficient adapter fine-tuning across a set of interconnected edge devices, forming a ring topology for per-batch training by sequentially placing frozen transformer blocks and their trainable adapter modules on the devices. RingAda follows a novel pipeline-parallel training mechanism with top-down adapter unfreezing, allowing for early-stopping of backpropagation at the lowest unfrozen adapter layer, thereby accelerating the fine-tuning process. Extensive experimental results demonstrate that RingAda significantly reduces fine-tuning time and memory costs while maintaining competitive model performance compared to its peer designs. |
| title | RingAda: Pipelining Large Model Fine-Tuning on Edge Devices with Scheduled Layer Unfreezing |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2502.19864 |