Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.05365 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918486836510720 |
|---|---|
| author | Washbourne, Robert Iyer, Rishi Figliolia, Tomas Zheng, Henry Lorig-Roach, Ryan Yang, Sungyeon Yuvraj, Pritish Anthony, Quentin Tokpanov, Yury Yang, Xiao Nanduru, Ganesh Ebert, Stephen Medepalli, Praneeth Szot, Skyler Rajagopal, Srivatsan Ong, Alex Mehta, Bhavana Millidge, Beren |
| author_facet | Washbourne, Robert Iyer, Rishi Figliolia, Tomas Zheng, Henry Lorig-Roach, Ryan Yang, Sungyeon Yuvraj, Pritish Anthony, Quentin Tokpanov, Yury Yang, Xiao Nanduru, Ganesh Ebert, Stephen Medepalli, Praneeth Szot, Skyler Rajagopal, Srivatsan Ong, Alex Mehta, Bhavana Millidge, Beren |
| contents | We present ZAYA1-8B, a reasoning-focused mixture-of-experts (MoE) model with 700M active and 8B total parameters, built on Zyphra's MoE++ architecture. ZAYA1-8B's core pretraining, midtraining, and supervised fine-tuning (SFT) were performed on a full-stack AMD compute, networking, and software platform. With under 1B active parameters, ZAYA1-8B matches or exceeds DeepSeek-R1-0528 on several challenging mathematics and coding benchmarks, and remains competitive with substantially larger open-weight reasoning models. ZAYA1-8B was trained from scratch for reasoning, with reasoning data included from pretraining onward using an answer-preserving trimming scheme. Post-training uses a four-stage RL cascade: reasoning warmup on math and puzzles; a 400-task RLVE-Gym curriculum; math and code RL with test-time compute traces and synthetic code environments built from competitive-programming references; and behavioral RL for chat and instruction following. We also introduce Markovian RSA, a test-time compute method that recursively aggregates parallel reasoning traces while carrying forward only bounded-length reasoning tails between rounds. In TTC evaluation, Markovian RSA raises ZAYA1-8B to 91.9\% on AIME'25 and 89.6\% on HMMT'25 while carrying forward only a 4K-token tail, narrowing the gap to much larger reasoning models including Gemini-2.5 Pro, DeepSeek-V3.2, and GPT-5-High. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_05365 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ZAYA1-8B Technical Report Washbourne, Robert Iyer, Rishi Figliolia, Tomas Zheng, Henry Lorig-Roach, Ryan Yang, Sungyeon Yuvraj, Pritish Anthony, Quentin Tokpanov, Yury Yang, Xiao Nanduru, Ganesh Ebert, Stephen Medepalli, Praneeth Szot, Skyler Rajagopal, Srivatsan Ong, Alex Mehta, Bhavana Millidge, Beren Artificial Intelligence Computation and Language We present ZAYA1-8B, a reasoning-focused mixture-of-experts (MoE) model with 700M active and 8B total parameters, built on Zyphra's MoE++ architecture. ZAYA1-8B's core pretraining, midtraining, and supervised fine-tuning (SFT) were performed on a full-stack AMD compute, networking, and software platform. With under 1B active parameters, ZAYA1-8B matches or exceeds DeepSeek-R1-0528 on several challenging mathematics and coding benchmarks, and remains competitive with substantially larger open-weight reasoning models. ZAYA1-8B was trained from scratch for reasoning, with reasoning data included from pretraining onward using an answer-preserving trimming scheme. Post-training uses a four-stage RL cascade: reasoning warmup on math and puzzles; a 400-task RLVE-Gym curriculum; math and code RL with test-time compute traces and synthetic code environments built from competitive-programming references; and behavioral RL for chat and instruction following. We also introduce Markovian RSA, a test-time compute method that recursively aggregates parallel reasoning traces while carrying forward only bounded-length reasoning tails between rounds. In TTC evaluation, Markovian RSA raises ZAYA1-8B to 91.9\% on AIME'25 and 89.6\% on HMMT'25 while carrying forward only a 4K-token tail, narrowing the gap to much larger reasoning models including Gemini-2.5 Pro, DeepSeek-V3.2, and GPT-5-High. |
| title | ZAYA1-8B Technical Report |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2605.05365 |