Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908403506348032 |
|---|---|
| author | Li, Pengyi Skripkin, Matvey Zubrey, Alexander Kuznetsov, Andrey Oseledets, Ivan |
| author_facet | Li, Pengyi Skripkin, Matvey Zubrey, Alexander Kuznetsov, Andrey Oseledets, Ivan |
| contents | Large language models (LLMs) excel at reasoning, yet post-training remains critical for aligning their behavior with task goals. Existing reinforcement learning (RL) methods often depend on costly human annotations or external reward models. We propose Reinforcement Learning via Self-Confidence (RLSC), which uses the model's own confidence as reward signals-eliminating the need for labels, preference models, or reward engineering. Applied to Qwen2.5-Math-7B with only 16 samples per question and 10 or 20 training steps, RLSC improves accuracy by +13.4% on AIME2024, +21.2% on MATH500, +21.7% on Minerva Math, +20.8% on Olympiadbench, and +9.7% on AMC23. RLSC provides a simple, scalable post-training method for inference models, requiring only a small number of samples and unlabelled supervision. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_06395 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models Li, Pengyi Skripkin, Matvey Zubrey, Alexander Kuznetsov, Andrey Oseledets, Ivan Computation and Language Machine Learning Large language models (LLMs) excel at reasoning, yet post-training remains critical for aligning their behavior with task goals. Existing reinforcement learning (RL) methods often depend on costly human annotations or external reward models. We propose Reinforcement Learning via Self-Confidence (RLSC), which uses the model's own confidence as reward signals-eliminating the need for labels, preference models, or reward engineering. Applied to Qwen2.5-Math-7B with only 16 samples per question and 10 or 20 training steps, RLSC improves accuracy by +13.4% on AIME2024, +21.2% on MATH500, +21.7% on Minerva Math, +20.8% on Olympiadbench, and +9.7% on AMC23. RLSC provides a simple, scalable post-training method for inference models, requiring only a small number of samples and unlabelled supervision. |
| title | Confidence Is All You Need: Few-Shot RL Fine-Tuning of Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2506.06395 |