Flextron: Many-in-One Flexible Large Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917760650444800 |
|---|---|
| author | Cai, Ruisi Muralidharan, Saurav Heinrich, Greg Yin, Hongxu Wang, Zhangyang Kautz, Jan Molchanov, Pavlo |
| author_facet | Cai, Ruisi Muralidharan, Saurav Heinrich, Greg Yin, Hongxu Wang, Zhangyang Kautz, Jan Molchanov, Pavlo |
| contents | Training modern LLMs is extremely resource intensive, and customizing them for various deployment scenarios characterized by limited compute and memory resources through repeated training is impractical. In this paper, we introduce Flextron, a network architecture and post-training model optimization framework supporting flexible model deployment. The Flextron architecture utilizes a nested elastic structure to rapidly adapt to specific user-defined latency and accuracy targets during inference with no additional fine-tuning required. It is also input-adaptive, and can automatically route tokens through its sub-networks for improved performance and efficiency. We present a sample-efficient training method and associated routing algorithms for systematically transforming an existing trained LLM into a Flextron model. We evaluate Flextron on the GPT-3 and LLama-2 family of LLMs, and demonstrate superior performance over multiple end-to-end trained variants and other state-of-the-art elastic networks, all with a single pretraining run that consumes a mere 7.63% tokens compared to original pretraining. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10260 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Flextron: Many-in-One Flexible Large Language Model Cai, Ruisi Muralidharan, Saurav Heinrich, Greg Yin, Hongxu Wang, Zhangyang Kautz, Jan Molchanov, Pavlo Computation and Language Machine Learning Training modern LLMs is extremely resource intensive, and customizing them for various deployment scenarios characterized by limited compute and memory resources through repeated training is impractical. In this paper, we introduce Flextron, a network architecture and post-training model optimization framework supporting flexible model deployment. The Flextron architecture utilizes a nested elastic structure to rapidly adapt to specific user-defined latency and accuracy targets during inference with no additional fine-tuning required. It is also input-adaptive, and can automatically route tokens through its sub-networks for improved performance and efficiency. We present a sample-efficient training method and associated routing algorithms for systematically transforming an existing trained LLM into a Flextron model. We evaluate Flextron on the GPT-3 and LLama-2 family of LLMs, and demonstrate superior performance over multiple end-to-end trained variants and other state-of-the-art elastic networks, all with a single pretraining run that consumes a mere 7.63% tokens compared to original pretraining. |
| title | Flextron: Many-in-One Flexible Large Language Model |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2406.10260 |