SPICE: Self-Play In Corpus Environments Improves Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909874014650368 |
|---|---|
| author | Liu, Bo Jin, Chuanyang Kim, Seungone Yuan, Weizhe Zhao, Wenting Kulikov, Ilia Li, Xian Sukhbaatar, Sainbayar Lanchantin, Jack Weston, Jason |
| author_facet | Liu, Bo Jin, Chuanyang Kim, Seungone Yuan, Weizhe Zhao, Wenting Kulikov, Ilia Li, Xian Sukhbaatar, Sainbayar Lanchantin, Jack Weston, Jason |
| contents | Self-improving systems require environmental interaction for continuous adaptation. We introduce SPICE (Self-Play In Corpus Environments), a reinforcement learning framework where a single model acts in two roles: a Challenger that mines documents from a large corpus to generate diverse reasoning tasks, and a Reasoner that solves them. Through adversarial dynamics, the Challenger creates an automatic curriculum at the frontier of the Reasoner's capability, while corpus grounding provides the rich, near-inexhaustible external signal necessary for sustained improvement. Unlike existing ungrounded self-play methods that offer more limited benefits, SPICE achieves consistent gains across mathematical (+8.9%) and general reasoning (+9.8%) benchmarks on multiple model families. Our analysis reveals how document grounding is a key ingredient in SPICE to continuously generate its own increasingly challenging goals and achieve them, enabling sustained self-improvement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_24684 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SPICE: Self-Play In Corpus Environments Improves Reasoning Liu, Bo Jin, Chuanyang Kim, Seungone Yuan, Weizhe Zhao, Wenting Kulikov, Ilia Li, Xian Sukhbaatar, Sainbayar Lanchantin, Jack Weston, Jason Computation and Language Self-improving systems require environmental interaction for continuous adaptation. We introduce SPICE (Self-Play In Corpus Environments), a reinforcement learning framework where a single model acts in two roles: a Challenger that mines documents from a large corpus to generate diverse reasoning tasks, and a Reasoner that solves them. Through adversarial dynamics, the Challenger creates an automatic curriculum at the frontier of the Reasoner's capability, while corpus grounding provides the rich, near-inexhaustible external signal necessary for sustained improvement. Unlike existing ungrounded self-play methods that offer more limited benefits, SPICE achieves consistent gains across mathematical (+8.9%) and general reasoning (+9.8%) benchmarks on multiple model families. Our analysis reveals how document grounding is a key ingredient in SPICE to continuously generate its own increasingly challenging goals and achieve them, enabling sustained self-improvement. |
| title | SPICE: Self-Play In Corpus Environments Improves Reasoning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.24684 |