Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911126504079360 |
|---|---|
| author | Hwang, Dongyoon Lee, Hojoon Choo, Jaegul Park, Dongmin Park, Jongho |
| author_facet | Hwang, Dongyoon Lee, Hojoon Choo, Jaegul Park, Dongmin Park, Jongho |
| contents | While reinforcement learning (RL) for large language models (LLMs) has shown promise in mathematical reasoning, strategic reasoning for LLMs using RL remains largely unexplored. We investigate whether LLMs can develop strategic reasoning capabilities through RL in chess. To this end, we leverage a chess-pretrained action-value network to provide dense reward on the LLM's output move quality, which can be seen as a form of knowledge distillation. Our experiments show that our distillation-based dense rewards often outperform sparse binary rewards. However, surprisingly, all models plateau far below expert levels. We provide SFT and RL ablations on chess reasoning training and find evidence that this limitation stems from a deficit in the pretrained models' internal understanding of chess-a deficit which RL alone may not be able to fully overcome. The code is available at https://github.com/krafton-ai/Chess-R1. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_00726 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess Hwang, Dongyoon Lee, Hojoon Choo, Jaegul Park, Dongmin Park, Jongho Artificial Intelligence Machine Learning While reinforcement learning (RL) for large language models (LLMs) has shown promise in mathematical reasoning, strategic reasoning for LLMs using RL remains largely unexplored. We investigate whether LLMs can develop strategic reasoning capabilities through RL in chess. To this end, we leverage a chess-pretrained action-value network to provide dense reward on the LLM's output move quality, which can be seen as a form of knowledge distillation. Our experiments show that our distillation-based dense rewards often outperform sparse binary rewards. However, surprisingly, all models plateau far below expert levels. We provide SFT and RL ablations on chess reasoning training and find evidence that this limitation stems from a deficit in the pretrained models' internal understanding of chess-a deficit which RL alone may not be able to fully overcome. The code is available at https://github.com/krafton-ai/Chess-R1. |
| title | Can Large Language Models Develop Strategic Reasoning? Post-training Insights from Learning Chess |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2507.00726 |