Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915190031777792 |
|---|---|
| author | Jamialahmadi, Benyamin Kavehzadeh, Parsa Rezagholizadeh, Mehdi Farinneya, Parsa Rajabzadeh, Hossein Jafari, Aref Chen, Boxing Tahaei, Marzieh S. |
| author_facet | Jamialahmadi, Benyamin Kavehzadeh, Parsa Rezagholizadeh, Mehdi Farinneya, Parsa Rajabzadeh, Hossein Jafari, Aref Chen, Boxing Tahaei, Marzieh S. |
| contents | Deploying large language models (LLMs) in real-world applications is often hindered by strict computational and latency constraints. While dynamic inference offers the flexibility to adjust model behavior based on varying resource budgets, existing methods are frequently limited by hardware inefficiencies or performance degradation. In this paper, we introduce Balcony, a simple yet highly effective framework for depth-based dynamic inference. By freezing the pretrained LLM and inserting additional transformer layers at selected exit points, Balcony maintains the full model's performance while enabling real-time adaptation to different computational budgets. These additional layers are trained using a straightforward self-distillation loss, aligning the sub-model outputs with those of the full model. This approach requires significantly fewer training tokens and tunable parameters, drastically reducing computational costs compared to prior methods. When applied to the LLaMA3-8B model, using only 0.2% of the original pretraining data, Balcony achieves minimal performance degradation while enabling significant speedups. Remarkably, we show that Balcony outperforms state-of-the-art methods such as Flextron and Layerskip as well as other leading compression techniques on multiple models and at various scales, across a variety of benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_05005 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models Jamialahmadi, Benyamin Kavehzadeh, Parsa Rezagholizadeh, Mehdi Farinneya, Parsa Rajabzadeh, Hossein Jafari, Aref Chen, Boxing Tahaei, Marzieh S. Computation and Language Artificial Intelligence Machine Learning Deploying large language models (LLMs) in real-world applications is often hindered by strict computational and latency constraints. While dynamic inference offers the flexibility to adjust model behavior based on varying resource budgets, existing methods are frequently limited by hardware inefficiencies or performance degradation. In this paper, we introduce Balcony, a simple yet highly effective framework for depth-based dynamic inference. By freezing the pretrained LLM and inserting additional transformer layers at selected exit points, Balcony maintains the full model's performance while enabling real-time adaptation to different computational budgets. These additional layers are trained using a straightforward self-distillation loss, aligning the sub-model outputs with those of the full model. This approach requires significantly fewer training tokens and tunable parameters, drastically reducing computational costs compared to prior methods. When applied to the LLaMA3-8B model, using only 0.2% of the original pretraining data, Balcony achieves minimal performance degradation while enabling significant speedups. Remarkably, we show that Balcony outperforms state-of-the-art methods such as Flextron and Layerskip as well as other leading compression techniques on multiple models and at various scales, across a variety of benchmarks. |
| title | Balcony: A Lightweight Approach to Dynamic Inference of Generative Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2503.05005 |