Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914362972700672 |
|---|---|
| author | Lys, Jonathan Gripon, Vincent Pasdeloup, Bastien Marmoret, Axel Mauch, Lukas Cardinaux, Fabien Hacene, Ghouthi Boukli |
| author_facet | Lys, Jonathan Gripon, Vincent Pasdeloup, Bastien Marmoret, Axel Mauch, Lukas Cardinaux, Fabien Hacene, Ghouthi Boukli |
| contents | Deep Learning architectures, and in particular Transformers, are conventionally viewed as a composition of layers. These layers are actually often obtained as the sum of two contributions: a residual path that copies the input and the output of a Transformer block. As a consequence, the inner representations (i.e. the input of these blocks) can be interpreted as iterative refinement of a propagated latent representation. Under this lens, many works suggest that the inner space is shared across layers, meaning that tokens can be decoded at early stages. Mechanistic interpretability even goes further by conjecturing that some layers act as refinement layers. Following this path, we propose inference-time inner looping, which prolongs refinement in pretrained off-the-shelf language models by repeatedly re-applying a selected block range. Across multiple benchmarks, inner looping yields modest but consistent accuracy improvements. Analyses of the resulting latent trajectories suggest more stable state evolution and continued semantic refinement. Overall, our results suggest that additional refinement can be obtained through simple test-time looping, extending computation in frozen pretrained models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_14759 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training Lys, Jonathan Gripon, Vincent Pasdeloup, Bastien Marmoret, Axel Mauch, Lukas Cardinaux, Fabien Hacene, Ghouthi Boukli Machine Learning Artificial Intelligence Deep Learning architectures, and in particular Transformers, are conventionally viewed as a composition of layers. These layers are actually often obtained as the sum of two contributions: a residual path that copies the input and the output of a Transformer block. As a consequence, the inner representations (i.e. the input of these blocks) can be interpreted as iterative refinement of a propagated latent representation. Under this lens, many works suggest that the inner space is shared across layers, meaning that tokens can be decoded at early stages. Mechanistic interpretability even goes further by conjecturing that some layers act as refinement layers. Following this path, we propose inference-time inner looping, which prolongs refinement in pretrained off-the-shelf language models by repeatedly re-applying a selected block range. Across multiple benchmarks, inner looping yields modest but consistent accuracy improvements. Analyses of the resulting latent trajectories suggest more stable state evolution and continued semantic refinement. Overall, our results suggest that additional refinement can be obtained through simple test-time looping, extending computation in frozen pretrained models. |
| title | Inner Loop Inference for Pretrained Transformers: Unlocking Latent Capabilities Without Training |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.14759 |