vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909310180655104 |
|---|---|
| author | Bang, Jehyeon Choi, Yujeong Kim, Myeongwoo Kim, Yongdeok Rhu, Minsoo |
| author_facet | Bang, Jehyeon Choi, Yujeong Kim, Myeongwoo Kim, Yongdeok Rhu, Minsoo |
| contents | As large language models (LLMs) become widespread in various application domains, a critical challenge the AI community is facing is how to train these large AI models in a cost-effective manner. Existing LLM training plans typically employ a heuristic based parallel training strategy which is based on empirical observations rather than grounded upon a thorough examination of the search space of LLM parallelization. Such limitation renders existing systems to leave significant performance left on the table, wasting millions of dollars worth of training cost. This paper presents our profiling-driven simulator called vTrain, providing AI practitioners a fast yet accurate software framework to determine an efficient and cost-effective LLM training system configuration. We demonstrate vTrain's practicality through several case studies, e.g., effectively evaluating optimal training parallelization strategies that balances training time and its associated training cost, efficient multi-tenant GPU cluster schedulers targeting multiple LLM training jobs, and determining a compute-optimal LLM model architecture given a fixed compute budget. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_12391 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training Bang, Jehyeon Choi, Yujeong Kim, Myeongwoo Kim, Yongdeok Rhu, Minsoo Machine Learning Artificial Intelligence Hardware Architecture As large language models (LLMs) become widespread in various application domains, a critical challenge the AI community is facing is how to train these large AI models in a cost-effective manner. Existing LLM training plans typically employ a heuristic based parallel training strategy which is based on empirical observations rather than grounded upon a thorough examination of the search space of LLM parallelization. Such limitation renders existing systems to leave significant performance left on the table, wasting millions of dollars worth of training cost. This paper presents our profiling-driven simulator called vTrain, providing AI practitioners a fast yet accurate software framework to determine an efficient and cost-effective LLM training system configuration. We demonstrate vTrain's practicality through several case studies, e.g., effectively evaluating optimal training parallelization strategies that balances training time and its associated training cost, efficient multi-tenant GPU cluster schedulers targeting multiple LLM training jobs, and determining a compute-optimal LLM model architecture given a fixed compute budget. |
| title | vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training |
| topic | Machine Learning Artificial Intelligence Hardware Architecture |
| url | https://arxiv.org/abs/2312.12391 |