Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.09078 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910047251988480 |
|---|---|
| author | Zhai, Shuangfei |
| author_facet | Zhai, Shuangfei |
| contents | We introduce exclusive self attention (XSA), a simple modification of self attention (SA) that improves Transformer's sequence modeling performance. The key idea is to constrain attention to capture only information orthogonal to the token's own value vector (thus excluding information of self position), encouraging better context modeling. Evaluated on the standard language modeling task, XSA consistently outperforms SA across model sizes up to 2.7B parameters and shows increasingly larger gains as sequence length grows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_09078 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Exclusive Self Attention Zhai, Shuangfei Machine Learning Computation and Language We introduce exclusive self attention (XSA), a simple modification of self attention (SA) that improves Transformer's sequence modeling performance. The key idea is to constrain attention to capture only information orthogonal to the token's own value vector (thus excluding information of self position), encouraging better context modeling. Evaluated on the standard language modeling task, XSA consistently outperforms SA across model sizes up to 2.7B parameters and shows increasingly larger gains as sequence length grows. |
| title | Exclusive Self Attention |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2603.09078 |