Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908255534448640 |
|---|---|
| author | Ku, Jerome Nguyen, Eric Romero, David W. Brixi, Garyk Yang, Brandon Vorontsov, Anton Taghibakhshi, Ali Lu, Amy X. Burke, Dave P. Brockman, Greg Massaroli, Stefano Ré, Christopher Hsu, Patrick D. Hie, Brian L. Ermon, Stefano Poli, Michael |
| author_facet | Ku, Jerome Nguyen, Eric Romero, David W. Brixi, Garyk Yang, Brandon Vorontsov, Anton Taghibakhshi, Ali Lu, Amy X. Burke, Dave P. Brockman, Greg Massaroli, Stefano Ré, Christopher Hsu, Patrick D. Hie, Brian L. Ermon, Stefano Poli, Michael |
| contents | We introduce convolutional multi-hybrid architectures, with a design grounded on two simple observations. First, operators in hybrid models can be tailored to token manipulation tasks such as in-context recall, multi-token recall, and compression, with input-dependent convolutions and attention offering complementary performance. Second, co-designing convolution operators and hardware-aware algorithms enables efficiency gains in regimes where previous alternative architectures struggle to surpass Transformers. At the 40 billion parameter scale, we train end-to-end 1.2 to 2.9 times faster than optimized Transformers, and 1.1 to 1.4 times faster than previous generation hybrids. On H100 GPUs and model width 4096, individual operators in the proposed multi-hybrid StripedHyena 2 architecture achieve two-fold throughput improvement over linear attention and state-space models. Multi-hybrids excel at sequence modeling over byte-tokenized data, as demonstrated by the Evo 2 line of models. We discuss the foundations that enable these results, including architecture design, overlap-add blocked kernels for tensor cores, and dedicated all-to-all and point-to-point context parallelism strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_01868 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale Ku, Jerome Nguyen, Eric Romero, David W. Brixi, Garyk Yang, Brandon Vorontsov, Anton Taghibakhshi, Ali Lu, Amy X. Burke, Dave P. Brockman, Greg Massaroli, Stefano Ré, Christopher Hsu, Patrick D. Hie, Brian L. Ermon, Stefano Poli, Michael Machine Learning Artificial Intelligence Computation and Language Distributed, Parallel, and Cluster Computing We introduce convolutional multi-hybrid architectures, with a design grounded on two simple observations. First, operators in hybrid models can be tailored to token manipulation tasks such as in-context recall, multi-token recall, and compression, with input-dependent convolutions and attention offering complementary performance. Second, co-designing convolution operators and hardware-aware algorithms enables efficiency gains in regimes where previous alternative architectures struggle to surpass Transformers. At the 40 billion parameter scale, we train end-to-end 1.2 to 2.9 times faster than optimized Transformers, and 1.1 to 1.4 times faster than previous generation hybrids. On H100 GPUs and model width 4096, individual operators in the proposed multi-hybrid StripedHyena 2 architecture achieve two-fold throughput improvement over linear attention and state-space models. Multi-hybrids excel at sequence modeling over byte-tokenized data, as demonstrated by the Evo 2 line of models. We discuss the foundations that enable these results, including architecture design, overlap-add blocked kernels for tensor cores, and dedicated all-to-all and point-to-point context parallelism strategies. |
| title | Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale |
| topic | Machine Learning Artificial Intelligence Computation and Language Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2503.01868 |