AutoHete: An Automatic and Efficient Heterogeneous Training System for LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915181210107904 |
|---|---|
| author | Zeng, Zihao Liu, Chubo He, Xin Hu, Juan Jiang, Yong Huang, Fei Li, Kenli Lim, Wei Yang Bryan |
| author_facet | Zeng, Zihao Liu, Chubo He, Xin Hu, Juan Jiang, Yong Huang, Fei Li, Kenli Lim, Wei Yang Bryan |
| contents | Transformer-based large language models (LLMs) have demonstrated exceptional capabilities in sequence modeling and text generation, with improvements scaling proportionally with model size. However, the limitations of GPU memory have restricted LLM training accessibility for many researchers. Existing heterogeneous training methods significantly expand the scale of trainable models but introduce substantial communication overheads and CPU workloads. In this work, we propose AutoHete, an automatic and efficient heterogeneous training system compatible with both single-GPU and multi-GPU environments. AutoHete dynamically adjusts activation checkpointing, parameter offloading, and optimizer offloading based on the specific hardware configuration and LLM training needs. Additionally, we design a priority-based scheduling mechanism that maximizes the overlap between operations across training iterations, enhancing throughput. Compared to state-of-the-art heterogeneous training systems, AutoHete delivers a 1.32x~1.91x throughput improvement across various model sizes and training configurations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01890 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AutoHete: An Automatic and Efficient Heterogeneous Training System for LLMs Zeng, Zihao Liu, Chubo He, Xin Hu, Juan Jiang, Yong Huang, Fei Li, Kenli Lim, Wei Yang Bryan Machine Learning Transformer-based large language models (LLMs) have demonstrated exceptional capabilities in sequence modeling and text generation, with improvements scaling proportionally with model size. However, the limitations of GPU memory have restricted LLM training accessibility for many researchers. Existing heterogeneous training methods significantly expand the scale of trainable models but introduce substantial communication overheads and CPU workloads. In this work, we propose AutoHete, an automatic and efficient heterogeneous training system compatible with both single-GPU and multi-GPU environments. AutoHete dynamically adjusts activation checkpointing, parameter offloading, and optimizer offloading based on the specific hardware configuration and LLM training needs. Additionally, we design a priority-based scheduling mechanism that maximizes the overlap between operations across training iterations, enhancing throughput. Compared to state-of-the-art heterogeneous training systems, AutoHete delivers a 1.32x~1.91x throughput improvement across various model sizes and training configurations. |
| title | AutoHete: An Automatic and Efficient Heterogeneous Training System for LLMs |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.01890 |