Split Knowledge Distillation for Large Models in IoT: Architecture, Challenges, and Solutions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912208333570048 |
|---|---|
| author | Li, Zuguang Wu, Wen Wu, Shaohua Lin, Qiaohua Sun, Yaping Wang, Hui |
| author_facet | Li, Zuguang Wu, Wen Wu, Shaohua Lin, Qiaohua Sun, Yaping Wang, Hui |
| contents | Large models (LMs) have immense potential in Internet of Things (IoT) systems, enabling applications such as intelligent voice assistants, predictive maintenance, and healthcare monitoring. However, training LMs on edge servers raises data privacy concerns, while deploying them directly on IoT devices is constrained by limited computational and memory resources. We analyze the key challenges of training LMs in IoT systems, including energy constraints, latency requirements, and device heterogeneity, and propose potential solutions such as dynamic resource management, adaptive model partitioning, and clustered collaborative training. Furthermore, we propose a split knowledge distillation framework to efficiently distill LMs into smaller, deployable versions for IoT devices while ensuring raw data remains local. This framework integrates knowledge distillation and split learning to minimize energy consumption and meet low model training delay requirements. A case study is presented to evaluate the feasibility and performance of the proposed framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_17164 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Split Knowledge Distillation for Large Models in IoT: Architecture, Challenges, and Solutions Li, Zuguang Wu, Wen Wu, Shaohua Lin, Qiaohua Sun, Yaping Wang, Hui Machine Learning Artificial Intelligence Large models (LMs) have immense potential in Internet of Things (IoT) systems, enabling applications such as intelligent voice assistants, predictive maintenance, and healthcare monitoring. However, training LMs on edge servers raises data privacy concerns, while deploying them directly on IoT devices is constrained by limited computational and memory resources. We analyze the key challenges of training LMs in IoT systems, including energy constraints, latency requirements, and device heterogeneity, and propose potential solutions such as dynamic resource management, adaptive model partitioning, and clustered collaborative training. Furthermore, we propose a split knowledge distillation framework to efficiently distill LMs into smaller, deployable versions for IoT devices while ensuring raw data remains local. This framework integrates knowledge distillation and split learning to minimize energy consumption and meet low model training delay requirements. A case study is presented to evaluate the feasibility and performance of the proposed framework. |
| title | Split Knowledge Distillation for Large Models in IoT: Architecture, Challenges, and Solutions |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2501.17164 |