Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.20503 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909805325582336 |
|---|---|
| author | Egorov, Evgenii Ackermann, Hanno Nagel, Markus Cai, Hong |
| author_facet | Egorov, Evgenii Ackermann, Hanno Nagel, Markus Cai, Hong |
| contents | Attention layers apply a sequence-to-sequence mapping whose parameters depend on the pairwise interactions of the input elements. However, without any structural assumptions, memory and compute scale quadratically with the sequence length. The two main ways to mitigate this are to introduce sparsity by ignoring a sufficient amount of pairwise interactions or to introduce recurrent dependence along them, as SSM does. Although both approaches are reasonable, they both have disadvantages. We propose a novel algorithm that combines the advantages of both concepts. Our idea is based on the efficient inversion of tree-structured matrices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_20503 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Myosotis: structured computation for attention like layer Egorov, Evgenii Ackermann, Hanno Nagel, Markus Cai, Hong Machine Learning Attention layers apply a sequence-to-sequence mapping whose parameters depend on the pairwise interactions of the input elements. However, without any structural assumptions, memory and compute scale quadratically with the sequence length. The two main ways to mitigate this are to introduce sparsity by ignoring a sufficient amount of pairwise interactions or to introduce recurrent dependence along them, as SSM does. Although both approaches are reasonable, they both have disadvantages. We propose a novel algorithm that combines the advantages of both concepts. Our idea is based on the efficient inversion of tree-structured matrices. |
| title | Myosotis: structured computation for attention like layer |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.20503 |