Super Apriel: One Checkpoint, Many Speeds
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914497733591040 |
|---|---|
| author | Labs, SLAM : Ostapenko, Oleksiy Li, Raymond Scholak, Torsten Mousavi-Hosseini, Alireza Tiwari, Aman Kocetkov, Denis Poirier, Joel Lamy Ogueji, Kelechi Krishna, Nanda H Pardinas, Rafael Madhusudhan, Sathwik Tejaswi Radhakrishna, Shruthan Sunkara, Srinivas Becaert, Valerie |
| author_facet | Labs, SLAM : Ostapenko, Oleksiy Li, Raymond Scholak, Torsten Mousavi-Hosseini, Alireza Tiwari, Aman Kocetkov, Denis Poirier, Joel Lamy Ogueji, Kelechi Krishna, Nanda H Pardinas, Rafael Madhusudhan, Sathwik Tejaswi Radhakrishna, Shruthan Sunkara, Srinivas Becaert, Valerie |
| contents | We release Super Apriel, a 15B-parameter supernet in which every decoder layer provides four trained mixer choices -- Full Attention (FA), Sliding Window Attention (SWA), Kimi Delta Attention (KDA), and Gated DeltaNet (GDN). A placement selects one mixer per layer; placements can be switched between requests at serving time without reloading weights, enabling multiple speed presets from a single checkpoint. The shared checkpoint also enables speculative decoding without a separate draft model. The all-FA preset matches the Apriel 1.6 teacher on all reported benchmarks; recommended hybrid presets span $2.9\times$ to $10.7\times$ decode throughput at 96% to 77% quality retention, with throughput advantages that compound at longer context lengths. With four mixer types across 48 layers, the configuration space is vast. A surrogate that predicts placement quality from the per-layer mixer assignment makes the speed-quality landscape tractable and identifies the best tradeoffs at each speed level. We investigate whether the best configurations at each speed level can be identified early in training or only after convergence. Rankings stabilize quickly at 0.5B scale, but the most efficient configurations exhibit higher instability at 15B, cautioning against extrapolation from smaller models. Super Apriel is trained by stochastic distillation from a frozen Apriel 1.6 teacher, followed by supervised fine-tuning. We release the supernet weights, Fast-LLM training code, vLLM serving code, and a placement optimization toolkit. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_19877 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Super Apriel: One Checkpoint, Many Speeds Labs, SLAM : Ostapenko, Oleksiy Li, Raymond Scholak, Torsten Mousavi-Hosseini, Alireza Tiwari, Aman Kocetkov, Denis Poirier, Joel Lamy Ogueji, Kelechi Krishna, Nanda H Pardinas, Rafael Madhusudhan, Sathwik Tejaswi Radhakrishna, Shruthan Sunkara, Srinivas Becaert, Valerie Machine Learning I.2.6; I.2.7 We release Super Apriel, a 15B-parameter supernet in which every decoder layer provides four trained mixer choices -- Full Attention (FA), Sliding Window Attention (SWA), Kimi Delta Attention (KDA), and Gated DeltaNet (GDN). A placement selects one mixer per layer; placements can be switched between requests at serving time without reloading weights, enabling multiple speed presets from a single checkpoint. The shared checkpoint also enables speculative decoding without a separate draft model. The all-FA preset matches the Apriel 1.6 teacher on all reported benchmarks; recommended hybrid presets span $2.9\times$ to $10.7\times$ decode throughput at 96% to 77% quality retention, with throughput advantages that compound at longer context lengths. With four mixer types across 48 layers, the configuration space is vast. A surrogate that predicts placement quality from the per-layer mixer assignment makes the speed-quality landscape tractable and identifies the best tradeoffs at each speed level. We investigate whether the best configurations at each speed level can be identified early in training or only after convergence. Rankings stabilize quickly at 0.5B scale, but the most efficient configurations exhibit higher instability at 15B, cautioning against extrapolation from smaller models. Super Apriel is trained by stochastic distillation from a frozen Apriel 1.6 teacher, followed by supervised fine-tuning. We release the supernet weights, Fast-LLM training code, vLLM serving code, and a placement optimization toolkit. |
| title | Super Apriel: One Checkpoint, Many Speeds |
| topic | Machine Learning I.2.6; I.2.7 |
| url | https://arxiv.org/abs/2604.19877 |