DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914015262801920 |
|---|---|
| author | Sharma, Aman Najafi, Saeed Farinneya, Parsa Jamialahmadi, Benyamin Tahaei, Marzieh S. Fan, Yuhe Rezagholizadeh, Mehdi Chen, Boxing Jafari, Aref |
| author_facet | Sharma, Aman Najafi, Saeed Farinneya, Parsa Jamialahmadi, Benyamin Tahaei, Marzieh S. Fan, Yuhe Rezagholizadeh, Mehdi Chen, Boxing Jafari, Aref |
| contents | Transformers achieve state-of-the-art results across many tasks, but their uniform application of quadratic self-attention to every token at every layer makes them computationally expensive. We introduce DTRNet (Dynamic Token Routing Network), an improved Transformer architecture that allows tokens to dynamically skip the quadratic cost of cross-token mixing while still receiving lightweight linear updates. By preserving the MLP module and reducing the attention cost for most tokens to linear, DTRNet ensures that every token is explicitly updated while significantly lowering overall computation. This design offers an efficient and effective alternative to standard dense attention. Once trained, DTRNet blocks routes only ~10% of tokens through attention at each layer while maintaining performance comparable to a full Transformer. It consistently outperforms routing-based layer skipping methods such as MoD and D-LLM in both accuracy and memory at matched FLOPs, while routing fewer tokens to full attention. Its efficiency gains, scales with sequence length, offering significant reduction in FLOPs for long-context inputs. By decoupling token updates from attention mixing, DTRNet substantially reduces the quadratic share of computation, providing a simple, efficient, and scalable alternative to Transformers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_00925 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers Sharma, Aman Najafi, Saeed Farinneya, Parsa Jamialahmadi, Benyamin Tahaei, Marzieh S. Fan, Yuhe Rezagholizadeh, Mehdi Chen, Boxing Jafari, Aref Machine Learning Computation and Language Transformers achieve state-of-the-art results across many tasks, but their uniform application of quadratic self-attention to every token at every layer makes them computationally expensive. We introduce DTRNet (Dynamic Token Routing Network), an improved Transformer architecture that allows tokens to dynamically skip the quadratic cost of cross-token mixing while still receiving lightweight linear updates. By preserving the MLP module and reducing the attention cost for most tokens to linear, DTRNet ensures that every token is explicitly updated while significantly lowering overall computation. This design offers an efficient and effective alternative to standard dense attention. Once trained, DTRNet blocks routes only ~10% of tokens through attention at each layer while maintaining performance comparable to a full Transformer. It consistently outperforms routing-based layer skipping methods such as MoD and D-LLM in both accuracy and memory at matched FLOPs, while routing fewer tokens to full attention. Its efficiency gains, scales with sequence length, offering significant reduction in FLOPs for long-context inputs. By decoupling token updates from attention mixing, DTRNet substantially reduces the quadratic share of computation, providing a simple, efficient, and scalable alternative to Transformers. |
| title | DTRNet: Dynamic Token Routing Network to Reduce Quadratic Costs in Transformers |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2509.00925 |