PoM: A Linear-Time Replacement for Attention with the Polynomial Mixer
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914454226075648 |
|---|---|
| author | Picard, David Dufour, Nicolas Degeorge, Lucas Ghosh, Arijit Allegro, Davide Ravaud, Tom Perron, Yohann Sautier, Corentin Baltaci, Zeynep Sonat Meng, Fei Kalleli, Syrine López-Rauhut, Marta Loiseau, Thibaut Albouy, Ségolène Baena, Raphael Vincent, Elliot Landrieu, Loic |
| author_facet | Picard, David Dufour, Nicolas Degeorge, Lucas Ghosh, Arijit Allegro, Davide Ravaud, Tom Perron, Yohann Sautier, Corentin Baltaci, Zeynep Sonat Meng, Fei Kalleli, Syrine López-Rauhut, Marta Loiseau, Thibaut Albouy, Ségolène Baena, Raphael Vincent, Elliot Landrieu, Loic |
| contents | This paper introduces the Polynomial Mixer (PoM), a novel token mixing mechanism with linear complexity that serves as a drop-in replacement for self-attention. PoM aggregates input tokens into a compact representation through a learned polynomial function, from which each token retrieves contextual information. We prove that PoM satisfies the contextual mapping property, ensuring that transformers equipped with PoM remain universal sequence-to-sequence approximators. We replace standard self-attention with PoM across five diverse domains: text generation, handwritten text recognition, image generation, 3D modeling, and Earth observation. PoM matches the performance of attention-based models while drastically reducing computational cost when working with long sequences. The code is available at https://github.com/davidpicard/pom. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_06129 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PoM: A Linear-Time Replacement for Attention with the Polynomial Mixer Picard, David Dufour, Nicolas Degeorge, Lucas Ghosh, Arijit Allegro, Davide Ravaud, Tom Perron, Yohann Sautier, Corentin Baltaci, Zeynep Sonat Meng, Fei Kalleli, Syrine López-Rauhut, Marta Loiseau, Thibaut Albouy, Ségolène Baena, Raphael Vincent, Elliot Landrieu, Loic Computer Vision and Pattern Recognition Artificial Intelligence This paper introduces the Polynomial Mixer (PoM), a novel token mixing mechanism with linear complexity that serves as a drop-in replacement for self-attention. PoM aggregates input tokens into a compact representation through a learned polynomial function, from which each token retrieves contextual information. We prove that PoM satisfies the contextual mapping property, ensuring that transformers equipped with PoM remain universal sequence-to-sequence approximators. We replace standard self-attention with PoM across five diverse domains: text generation, handwritten text recognition, image generation, 3D modeling, and Earth observation. PoM matches the performance of attention-based models while drastically reducing computational cost when working with long sequences. The code is available at https://github.com/davidpicard/pom. |
| title | PoM: A Linear-Time Replacement for Attention with the Polynomial Mixer |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2604.06129 |