Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929734649118720 |
|---|---|
| author | Paliotta, Daniele Wang, Junxiong Pagliardini, Matteo Li, Kevin Y. Bick, Aviv Kolter, J. Zico Gu, Albert Fleuret, François Dao, Tri |
| author_facet | Paliotta, Daniele Wang, Junxiong Pagliardini, Matteo Li, Kevin Y. Bick, Aviv Kolter, J. Zico Gu, Albert Fleuret, François Dao, Tri |
| contents | Recent advancements have demonstrated that the performance of large language models (LLMs) can be significantly enhanced by scaling computational resources at test time. A common strategy involves generating multiple Chain-of-Thought (CoT) trajectories and aggregating their outputs through various selection mechanisms. This raises a fundamental question: can models with lower complexity leverage their superior generation throughput to outperform similarly sized Transformers for a fixed computational budget? To address this question and overcome the lack of strong subquadratic reasoners, we distill pure and hybrid Mamba models from pretrained Transformers. Trained on only 8 billion tokens, our distilled models show strong performance and scaling on mathematical reasoning datasets while being much faster at inference for large batches and long sequences. Despite the zero-shot performance hit due to distillation, both pure and hybrid Mamba models can scale their coverage and accuracy performance past their Transformer teacher models under fixed time budgets, opening a new direction for scaling inference compute. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20339 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners Paliotta, Daniele Wang, Junxiong Pagliardini, Matteo Li, Kevin Y. Bick, Aviv Kolter, J. Zico Gu, Albert Fleuret, François Dao, Tri Computation and Language Artificial Intelligence Recent advancements have demonstrated that the performance of large language models (LLMs) can be significantly enhanced by scaling computational resources at test time. A common strategy involves generating multiple Chain-of-Thought (CoT) trajectories and aggregating their outputs through various selection mechanisms. This raises a fundamental question: can models with lower complexity leverage their superior generation throughput to outperform similarly sized Transformers for a fixed computational budget? To address this question and overcome the lack of strong subquadratic reasoners, we distill pure and hybrid Mamba models from pretrained Transformers. Trained on only 8 billion tokens, our distilled models show strong performance and scaling on mathematical reasoning datasets while being much faster at inference for large batches and long sequences. Despite the zero-shot performance hit due to distillation, both pure and hybrid Mamba models can scale their coverage and accuracy performance past their Transformer teacher models under fixed time budgets, opening a new direction for scaling inference compute. |
| title | Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasoners |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.20339 |