StaRPO: Stability-Augmented Reinforcement Policy Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918438190972928 |
|---|---|
| author | Zhang, Jinghan Mo, Fengran Weerasooriya, Tharindu Cyril Dai, Ruimin Han, Xiaoyan Fu, Yanjie Wang, Dakuo Liu, Kunpeng |
| author_facet | Zhang, Jinghan Mo, Fengran Weerasooriya, Tharindu Cyril Dai, Ruimin Han, Xiaoyan Fu, Yanjie Wang, Dakuo Liu, Kunpeng |
| contents | Reinforcement learning (RL) is effective in enhancing the accuracy of large language models in complex reasoning tasks. Existing RL policy optimization frameworks rely on final-answer correctness as feedback signals and rarely capture the internal logical structure of the reasoning process. Consequently, the models would generate fluent and semantically relevant responses but logically inconsistent, structurally erratic, or redundant. To this end, we propose StaRPO, a stability-augmented reinforcement learning framework that explicitly incorporates reasoning stability into the optimization objective. Our StaRPO decomposes stability into two computable lightweight metrics: the Autocorrelation Function (ACF) to evaluate local step-to-step coherence, and Path Efficiency (PE) to evaluate global goal-directedness of the reasoning trajectory. These stability rewards are combined with task rewards to provide complementary and process-aware feedback. We validate the effectiveness of using ACF and PE rewards by showing their correlation with logic errors on two backbone models. Experiments on four reasoning benchmarks show that StaRPO consistently outperforms compared baselines and can enhance both final-answer accuracy and logical stability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08905 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | StaRPO: Stability-Augmented Reinforcement Policy Optimization Zhang, Jinghan Mo, Fengran Weerasooriya, Tharindu Cyril Dai, Ruimin Han, Xiaoyan Fu, Yanjie Wang, Dakuo Liu, Kunpeng Artificial Intelligence Machine Learning Reinforcement learning (RL) is effective in enhancing the accuracy of large language models in complex reasoning tasks. Existing RL policy optimization frameworks rely on final-answer correctness as feedback signals and rarely capture the internal logical structure of the reasoning process. Consequently, the models would generate fluent and semantically relevant responses but logically inconsistent, structurally erratic, or redundant. To this end, we propose StaRPO, a stability-augmented reinforcement learning framework that explicitly incorporates reasoning stability into the optimization objective. Our StaRPO decomposes stability into two computable lightweight metrics: the Autocorrelation Function (ACF) to evaluate local step-to-step coherence, and Path Efficiency (PE) to evaluate global goal-directedness of the reasoning trajectory. These stability rewards are combined with task rewards to provide complementary and process-aware feedback. We validate the effectiveness of using ACF and PE rewards by showing their correlation with logic errors on two backbone models. Experiments on four reasoning benchmarks show that StaRPO consistently outperforms compared baselines and can enhance both final-answer accuracy and logical stability. |
| title | StaRPO: Stability-Augmented Reinforcement Policy Optimization |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2604.08905 |