Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.13876 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914313279635456 |
|---|---|
| author | Laitenberger, Filipe Kopiczko, Dawid Snoek, Cees G. M. Asano, Yuki M. |
| author_facet | Laitenberger, Filipe Kopiczko, Dawid Snoek, Cees G. M. Asano, Yuki M. |
| contents | We introduce GateSkip, a simple residual-stream gating mechanism that enables token-wise layer skipping in decoder-only LMs. Each Attention/MLP branch is equipped with a sigmoid-linear gate that condenses the branch's output before it re-enters the residual stream. During inference we rank tokens by the gate values and skip low-importance ones using a per-layer budget. While early-exit or router-based Mixture-of-Depths models are known to be unstable and need extensive retraining, our smooth, differentiable gates fine-tune stably on top of pretrained models. On long-form reasoning, we save up to 15% compute while retaining over 90% of baseline accuracy. For increasingly larger models, this tradeoff improves drastically. On instruction-tuned models we see accuracy gains at full compute and match baseline quality near 50% savings. The learned gates give insight into transformer information flow (e.g., BOS tokens act as anchors), and the method combines easily with quantization, pruning, and self-speculative decoding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_13876 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | What Layers When: Learning to Skip Compute in LLMs with Residual Gates Laitenberger, Filipe Kopiczko, Dawid Snoek, Cees G. M. Asano, Yuki M. Computation and Language Artificial Intelligence We introduce GateSkip, a simple residual-stream gating mechanism that enables token-wise layer skipping in decoder-only LMs. Each Attention/MLP branch is equipped with a sigmoid-linear gate that condenses the branch's output before it re-enters the residual stream. During inference we rank tokens by the gate values and skip low-importance ones using a per-layer budget. While early-exit or router-based Mixture-of-Depths models are known to be unstable and need extensive retraining, our smooth, differentiable gates fine-tune stably on top of pretrained models. On long-form reasoning, we save up to 15% compute while retaining over 90% of baseline accuracy. For increasingly larger models, this tradeoff improves drastically. On instruction-tuned models we see accuracy gains at full compute and match baseline quality near 50% savings. The learned gates give insight into transformer information flow (e.g., BOS tokens act as anchors), and the method combines easily with quantization, pruning, and self-speculative decoding. |
| title | What Layers When: Learning to Skip Compute in LLMs with Residual Gates |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.13876 |