FLARE: Fast Low-rank Attention Routing Engine
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908800834863104 |
|---|---|
| author | Puri, Vedant Joglekar, Aditya Bandreddi, Sri Datta Ganesh Ferguson, Kevin Chen, Yu-hsuan Zhang, Yongjie Jessica Kara, Levent Burak |
| author_facet | Puri, Vedant Joglekar, Aditya Bandreddi, Sri Datta Ganesh Ferguson, Kevin Chen, Yu-hsuan Zhang, Yongjie Jessica Kara, Levent Burak |
| contents | The quadratic complexity of self-attention limits the scalability of transformers on long sequences. We introduce Fast Low-rank Attention Routing Engine (FLARE), a token-mixing operator that realizes low-rank attention by routing information through a small set of latent tokens. Each layer induces an input-input token mixing matrix of rank at most $M$ via a minimal encode-decode factorization implemented using only two standard scaled dot-product attention (SDPA) calls. Because the dominant ${O}(NM)$ computation is expressed purely in terms of standard SDPA, FLARE is compatible with fused attention kernels and avoids materializing $M\times N$ projection matrices. FLARE further assigns disjoint latent slices to each attention head, yielding a mixture of head-specific low-rank pathways. Empirically, FLARE scales to one-million-point unstructured meshes on a single GPU, achieves state-of-the-art accuracy on PDE surrogate benchmarks, and outperforms general-purpose efficient-attention methods on the Long Range Arena suite. We additionally release a large-scale additive manufacturing benchmark dataset. Our code is available at https://github.com/vpuri3/FLARE.py. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_12594 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FLARE: Fast Low-rank Attention Routing Engine Puri, Vedant Joglekar, Aditya Bandreddi, Sri Datta Ganesh Ferguson, Kevin Chen, Yu-hsuan Zhang, Yongjie Jessica Kara, Levent Burak Machine Learning The quadratic complexity of self-attention limits the scalability of transformers on long sequences. We introduce Fast Low-rank Attention Routing Engine (FLARE), a token-mixing operator that realizes low-rank attention by routing information through a small set of latent tokens. Each layer induces an input-input token mixing matrix of rank at most $M$ via a minimal encode-decode factorization implemented using only two standard scaled dot-product attention (SDPA) calls. Because the dominant ${O}(NM)$ computation is expressed purely in terms of standard SDPA, FLARE is compatible with fused attention kernels and avoids materializing $M\times N$ projection matrices. FLARE further assigns disjoint latent slices to each attention head, yielding a mixture of head-specific low-rank pathways. Empirically, FLARE scales to one-million-point unstructured meshes on a single GPU, achieves state-of-the-art accuracy on PDE surrogate benchmarks, and outperforms general-purpose efficient-attention methods on the Long Range Arena suite. We additionally release a large-scale additive manufacturing benchmark dataset. Our code is available at https://github.com/vpuri3/FLARE.py. |
| title | FLARE: Fast Low-rank Attention Routing Engine |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2508.12594 |