Learning to Reason Efficiently with Discounted Reinforcement Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910259462799360 |
|---|---|
| author | Ayoub, Alex Asadi, Kavosh Schuurmans, Dale Szepesvári, Csaba Bouyarmane, Karim |
| author_facet | Ayoub, Alex Asadi, Kavosh Schuurmans, Dale Szepesvári, Csaba Bouyarmane, Karim |
| contents | Large reasoning models (LRMs) often consume excessive tokens, inflating computational cost and latency. More broadly, in goal reaching sequential decision problems we often want to reach the goal quickly, and LRM reasoning can be viewed through this lens. We challenge the assumption that longer responses improve accuracy. By penalizing reasoning tokens using a discounted reinforcement learning setup (interpretable as a small token cost) and analyzing Blackwell optimality in restricted policy classes, we encourage concise yet accurate reasoning, analogous to preferring shorter successful trajectories in a stochastic shortest path problem. Experiments confirm our theoretical results that this approach shortens chains of thought while preserving accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_23486 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning to Reason Efficiently with Discounted Reinforcement Learning Ayoub, Alex Asadi, Kavosh Schuurmans, Dale Szepesvári, Csaba Bouyarmane, Karim Machine Learning Large reasoning models (LRMs) often consume excessive tokens, inflating computational cost and latency. More broadly, in goal reaching sequential decision problems we often want to reach the goal quickly, and LRM reasoning can be viewed through this lens. We challenge the assumption that longer responses improve accuracy. By penalizing reasoning tokens using a discounted reinforcement learning setup (interpretable as a small token cost) and analyzing Blackwell optimality in restricted policy classes, we encourage concise yet accurate reasoning, analogous to preferring shorter successful trajectories in a stochastic shortest path problem. Experiments confirm our theoretical results that this approach shortens chains of thought while preserving accuracy. |
| title | Learning to Reason Efficiently with Discounted Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.23486 |