Learning Robust Reasoning through Guided Adversarial Self-Play
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914296402804736 |
|---|---|
| author | Li, Shuozhe Tadiparthi, Vaishnav Lee, Kwonjoon Agarwal, Nakul Mahjoub, Hossein Nourkhiz Pari, Ehsan Moradi Chen, Lizhang Zhang, Amy Leqi, Liu |
| author_facet | Li, Shuozhe Tadiparthi, Vaishnav Lee, Kwonjoon Agarwal, Nakul Mahjoub, Hossein Nourkhiz Pari, Ehsan Moradi Chen, Lizhang Zhang, Amy Leqi, Liu |
| contents | Reinforcement learning from verifiable rewards (RLVR) produces strong reasoning models, yet they can fail catastrophically when the conditioning context is fallible (e.g., corrupted chain-of-thought, misleading partial solutions, or mild input perturbations), since standard RLVR optimizes final-answer correctness only under clean conditioning. We introduce GASP (Guided Adversarial Self-Play), a robustification method that explicitly trains detect-and-repair capabilities using only outcome verification. Without human labels or external teachers, GASP forms an adversarial self-play game within a single model: a polluter learns to induce failure via locally coherent corruptions, while an agent learns to diagnose and recover under the same corrupted conditioning. To address the scarcity of successful recoveries early in training, we propose in-distribution repair guidance, an imitation term on self-generated repairs that increases recovery probability while preserving previously acquired capabilities. Across four open-weight models (1.5B--8B), GASP transforms strong-but-brittle reasoners into robust ones that withstand misleading and perturbed context while often improving clean accuracy. Further analysis shows that adversarial corruptions induce an effective curriculum, and in-distribution guidance enables rapid recovery learning with minimal representational drift. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_00173 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Learning Robust Reasoning through Guided Adversarial Self-Play Li, Shuozhe Tadiparthi, Vaishnav Lee, Kwonjoon Agarwal, Nakul Mahjoub, Hossein Nourkhiz Pari, Ehsan Moradi Chen, Lizhang Zhang, Amy Leqi, Liu Machine Learning Artificial Intelligence Reinforcement learning from verifiable rewards (RLVR) produces strong reasoning models, yet they can fail catastrophically when the conditioning context is fallible (e.g., corrupted chain-of-thought, misleading partial solutions, or mild input perturbations), since standard RLVR optimizes final-answer correctness only under clean conditioning. We introduce GASP (Guided Adversarial Self-Play), a robustification method that explicitly trains detect-and-repair capabilities using only outcome verification. Without human labels or external teachers, GASP forms an adversarial self-play game within a single model: a polluter learns to induce failure via locally coherent corruptions, while an agent learns to diagnose and recover under the same corrupted conditioning. To address the scarcity of successful recoveries early in training, we propose in-distribution repair guidance, an imitation term on self-generated repairs that increases recovery probability while preserving previously acquired capabilities. Across four open-weight models (1.5B--8B), GASP transforms strong-but-brittle reasoners into robust ones that withstand misleading and perturbed context while often improving clean accuracy. Further analysis shows that adversarial corruptions induce an effective curriculum, and in-distribution guidance enables rapid recovery learning with minimal representational drift. |
| title | Learning Robust Reasoning through Guided Adversarial Self-Play |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.00173 |