Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.15992 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915560947712000 |
|---|---|
| author | Dai, Ziming Zhang, Tuo Gao, Fei Cai, Xingyi Wang, Xiaofei Zhang, Cheng Wang, Wenyu Zang, Chengjie |
| author_facet | Dai, Ziming Zhang, Tuo Gao, Fei Cai, Xingyi Wang, Xiaofei Zhang, Cheng Wang, Wenyu Zang, Chengjie |
| contents | The growing industrial demand for customized and cost-efficient large language models (LLMs) is fueled by the rise of vertical, domain-specific tasks and the need to optimize performance under constraints such as latency and budget. Knowledge distillation, as an efficient model compression and transfer technique, offers a feasible solution. However, existing distillation frameworks often require manual intervention and struggle to meet such complex user-defined distillation requirements. To bridge this gap, we propose Stratos, an end-to-end LLM distillation pipeline that automates server and model selection, knowledge distillation, and deployment in distributed cloud environments. Given user-defined constraints on model performance and system budget, Stratos automatically selects Pareto-optimal servers, dynamically matches teacher-student pairs, and adapts distillation strategies based on task complexity to optimize cloud hosting. Experiments show that Stratos produces a student model that achieves four times the accuracy of its GPT-4o teacher baseline on a rare, domain-specific Mahjong reasoning task with reverse synthetic data and knowledge injection. Moreover, it achieves reduced latency and cost without compromising accuracy. These results highlight its promise for vertical-domain LLM deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_15992 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Stratos: An End-to-End Distillation Pipeline for Customized LLMs under Distributed Cloud Environments Dai, Ziming Zhang, Tuo Gao, Fei Cai, Xingyi Wang, Xiaofei Zhang, Cheng Wang, Wenyu Zang, Chengjie Machine Learning Artificial Intelligence The growing industrial demand for customized and cost-efficient large language models (LLMs) is fueled by the rise of vertical, domain-specific tasks and the need to optimize performance under constraints such as latency and budget. Knowledge distillation, as an efficient model compression and transfer technique, offers a feasible solution. However, existing distillation frameworks often require manual intervention and struggle to meet such complex user-defined distillation requirements. To bridge this gap, we propose Stratos, an end-to-end LLM distillation pipeline that automates server and model selection, knowledge distillation, and deployment in distributed cloud environments. Given user-defined constraints on model performance and system budget, Stratos automatically selects Pareto-optimal servers, dynamically matches teacher-student pairs, and adapts distillation strategies based on task complexity to optimize cloud hosting. Experiments show that Stratos produces a student model that achieves four times the accuracy of its GPT-4o teacher baseline on a rare, domain-specific Mahjong reasoning task with reverse synthetic data and knowledge injection. Moreover, it achieves reduced latency and cost without compromising accuracy. These results highlight its promise for vertical-domain LLM deployment. |
| title | Stratos: An End-to-End Distillation Pipeline for Customized LLMs under Distributed Cloud Environments |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.15992 |