Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.14685 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909028682039296 |
|---|---|
| author | Naim, Omar Bhar, Swarnadeep Bolte, Jérôme Asher, Nicholas |
| author_facet | Naim, Omar Bhar, Swarnadeep Bolte, Jérôme Asher, Nicholas |
| contents | While transformer models exhibit strong in-context learning (ICL) abilities, they often fail to generalize under simple distribution shifts. We analyze these failures and identify Softmax, the scoring function in the attention mechanism, as a contributing factor. We propose \textbf{Scaled Signed Averaging (SSA)}, a novel attention scoring function that mitigates these failures. SSA significantly improves performance on our ICL tasks and outperforms transformer models with Softmax on several NLP benchmarks and linguistic probing tasks, in both decoder-only and encoder-only architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_14685 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SSA: Improving Performance With a Better Scoring Function Naim, Omar Bhar, Swarnadeep Bolte, Jérôme Asher, Nicholas Computation and Language While transformer models exhibit strong in-context learning (ICL) abilities, they often fail to generalize under simple distribution shifts. We analyze these failures and identify Softmax, the scoring function in the attention mechanism, as a contributing factor. We propose \textbf{Scaled Signed Averaging (SSA)}, a novel attention scoring function that mitigates these failures. SSA significantly improves performance on our ICL tasks and outperforms transformer models with Softmax on several NLP benchmarks and linguistic probing tasks, in both decoder-only and encoder-only architectures. |
| title | SSA: Improving Performance With a Better Scoring Function |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.14685 |