Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914489545261056 |
|---|---|
| author | Ildiz, Muhammed Emrullah Gozeten, Halil Alperen Taga, Ege Onur Oymak, Samet |
| author_facet | Ildiz, Muhammed Emrullah Gozeten, Halil Alperen Taga, Ege Onur Oymak, Samet |
| contents | State-of-the-art reasoning models utilize long chain-of-thought (CoT) to solve increasingly complex problems using more test-time computation. In this work, we explore a long CoT setting where the model makes up to K successive attempts at solving a problem, in which each attempt is allowed to build on earlier ones after the model receives a hard verifier feedback. This motivates RL methods that can harness per-attempt rewards by carefully weighting individual attempts. We study optimizing the Verification@K reward (the model succeeds by the K-th attempt) and show that naively weighing the attempts by their pass/fail results in biased gradients. We introduce Calibrated Attempt-Level (CAL) GRPO by devising a weighing strategy to obtain unbiased gradients while maintaining small variance. Our theory reveals how incorporating per-attempt rewards influence the training and the eventual Verification@K performance. Experiments, baselines, and ablations on synthetic and real data corroborate our theory and the benefits of CAL-GRPO over vanilla GRPO as well as naive weighting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_17912 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought Ildiz, Muhammed Emrullah Gozeten, Halil Alperen Taga, Ege Onur Oymak, Samet Machine Learning Artificial Intelligence State-of-the-art reasoning models utilize long chain-of-thought (CoT) to solve increasingly complex problems using more test-time computation. In this work, we explore a long CoT setting where the model makes up to K successive attempts at solving a problem, in which each attempt is allowed to build on earlier ones after the model receives a hard verifier feedback. This motivates RL methods that can harness per-attempt rewards by carefully weighting individual attempts. We study optimizing the Verification@K reward (the model succeeds by the K-th attempt) and show that naively weighing the attempts by their pass/fail results in biased gradients. We introduce Calibrated Attempt-Level (CAL) GRPO by devising a weighing strategy to obtain unbiased gradients while maintaining small variance. Our theory reveals how incorporating per-attempt rewards influence the training and the eventual Verification@K performance. Experiments, baselines, and ablations on synthetic and real data corroborate our theory and the benefits of CAL-GRPO over vanilla GRPO as well as naive weighting. |
| title | Learning to Correct: Calibrated Reinforcement Learning for Multi-Attempt Chain-of-Thought |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2604.17912 |