LayerNorm Induces Recency Bias in Transformer Decoders
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913046715170816 |
|---|---|
| author | Kim, Junu Liu, Xiao Lin, Zhenghao Ji, Lei Gong, Yeyun Choi, Edward |
| author_facet | Kim, Junu Liu, Xiao Lin, Zhenghao Ji, Lei Gong, Yeyun Choi, Edward |
| contents | Causal self-attention provides positional information to Transformer decoders. Prior work has shown that stacks of causal self-attention layers alone induce a positional bias in attention scores toward earlier tokens. However, this differs from the bias toward later tokens typically observed in Transformer decoders, known as recency bias. We address this discrepancy by analyzing the interaction between causal self-attention and other architectural components. We show that stacked causal self-attention layers combined with LayerNorm induce recency bias. Furthermore, we examine the effects of residual connections and the distribution of input token embeddings on this bias. Our results provide new theoretical insights into how positional information interacts with architectural components and suggest directions for improving positional encoding strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_21042 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LayerNorm Induces Recency Bias in Transformer Decoders Kim, Junu Liu, Xiao Lin, Zhenghao Ji, Lei Gong, Yeyun Choi, Edward Computation and Language Machine Learning Causal self-attention provides positional information to Transformer decoders. Prior work has shown that stacks of causal self-attention layers alone induce a positional bias in attention scores toward earlier tokens. However, this differs from the bias toward later tokens typically observed in Transformer decoders, known as recency bias. We address this discrepancy by analyzing the interaction between causal self-attention and other architectural components. We show that stacked causal self-attention layers combined with LayerNorm induce recency bias. Furthermore, we examine the effects of residual connections and the distribution of input token embeddings on this bias. Our results provide new theoretical insights into how positional information interacts with architectural components and suggest directions for improving positional encoding strategies. |
| title | LayerNorm Induces Recency Bias in Transformer Decoders |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2509.21042 |