Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908859939946496 |
|---|---|
| author | Lys, Jonathan Gripon, Vincent Pasdeloup, Bastien Marmoret, Axel Mauch, Lukas Cardinaux, Fabien Hacene, Ghouthi Boukli |
| author_facet | Lys, Jonathan Gripon, Vincent Pasdeloup, Bastien Marmoret, Axel Mauch, Lukas Cardinaux, Fabien Hacene, Ghouthi Boukli |
| contents | Large Language Models (LLMs) are trained with next-token prediction, implemented in autoregressive Transformers via causal masking for parallelism. This creates a subtle misalignment: residual connections tie activations to the current token, while supervision targets the next token, potentially propagating mismatched information if the current token is not the most informative for prediction. In this work, we empirically localize this input-output alignment shift in pretrained LLMs, using decoding trajectories over tied embedding spaces and similarity-based metrics. Our experiments reveal that the hidden token representations switch from input alignment to output alignment deep within the network. Motivated by this observation, we propose a lightweight residual-path mitigation based on residual attenuation, implemented either as a fixed-layer intervention or as a learnable gating mechanism. Experiments on multiple benchmarks show that these strategies alleviate the representation misalignment and yield improvements, providing an efficient and general architectural enhancement for autoregressive Transformers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_14760 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers Lys, Jonathan Gripon, Vincent Pasdeloup, Bastien Marmoret, Axel Mauch, Lukas Cardinaux, Fabien Hacene, Ghouthi Boukli Computation and Language Artificial Intelligence Large Language Models (LLMs) are trained with next-token prediction, implemented in autoregressive Transformers via causal masking for parallelism. This creates a subtle misalignment: residual connections tie activations to the current token, while supervision targets the next token, potentially propagating mismatched information if the current token is not the most informative for prediction. In this work, we empirically localize this input-output alignment shift in pretrained LLMs, using decoding trajectories over tied embedding spaces and similarity-based metrics. Our experiments reveal that the hidden token representations switch from input alignment to output alignment deep within the network. Motivated by this observation, we propose a lightweight residual-path mitigation based on residual attenuation, implemented either as a fixed-layer intervention or as a learnable gating mechanism. Experiments on multiple benchmarks show that these strategies alleviate the representation misalignment and yield improvements, providing an efficient and general architectural enhancement for autoregressive Transformers. |
| title | Residual Connections and the Causal Shift: Uncovering a Structural Misalignment in Transformers |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2602.14760 |