The Diminishing Returns of Early-Exit Decoding in Modern LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910070393012224 |
|---|---|
| author | Wei, Rui Du, Rui Yu, Hanfei Tiwari, Devesh Li, Jian Xu, Zhaozhuo Wang, Hao |
| author_facet | Wei, Rui Du, Rui Yu, Hanfei Tiwari, Devesh Li, Jian Xu, Zhaozhuo Wang, Hao |
| contents | In Large Language Model (LLM) inference, early-exit refers to stopping computation at an intermediate layer once the prediction is sufficiently confident, thereby reducing latency and cost. However, recent LLMs adopt improved pretraining recipes and architectures that reduce layer redundancy, potentially limiting early-exit opportunities. We re-evaluate layer-wise early-exit in modern LLMs and analyze how intermediate representations evolve during training. We introduce a metric to quantify a model's intrinsic suitability for early-exit and propose a benchmark for researchers to explore the potential early-exit benefits on different models and workloads. Our results show a diminishing trend in early-exit effectiveness across newer model generations. We further find that dense transformers generally offer greater early-exit potential than Mixture-of-Experts and State Space Models. In addition, larger models, particularly those with more than 20 billion parameters, and base pretrained models without specialized tuning tend to exhibit higher early-exit potential. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_23701 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | The Diminishing Returns of Early-Exit Decoding in Modern LLMs Wei, Rui Du, Rui Yu, Hanfei Tiwari, Devesh Li, Jian Xu, Zhaozhuo Wang, Hao Computation and Language Artificial Intelligence In Large Language Model (LLM) inference, early-exit refers to stopping computation at an intermediate layer once the prediction is sufficiently confident, thereby reducing latency and cost. However, recent LLMs adopt improved pretraining recipes and architectures that reduce layer redundancy, potentially limiting early-exit opportunities. We re-evaluate layer-wise early-exit in modern LLMs and analyze how intermediate representations evolve during training. We introduce a metric to quantify a model's intrinsic suitability for early-exit and propose a benchmark for researchers to explore the potential early-exit benefits on different models and workloads. Our results show a diminishing trend in early-exit effectiveness across newer model generations. We further find that dense transformers generally offer greater early-exit potential than Mixture-of-Experts and State Space Models. In addition, larger models, particularly those with more than 20 billion parameters, and base pretrained models without specialized tuning tend to exhibit higher early-exit potential. |
| title | The Diminishing Returns of Early-Exit Decoding in Modern LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2603.23701 |