Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917301738012672 |
|---|---|
| author | Zhang, Nonghai Ma, Weitao Ma, Zhanyu Xu, Jun Gao, Jiuchong Hao, Jinghua He, Renqing Xu, Jingwen |
| author_facet | Zhang, Nonghai Ma, Weitao Ma, Zhanyu Xu, Jun Gao, Jiuchong Hao, Jinghua He, Renqing Xu, Jingwen |
| contents | Group Relative Policy Optimization (GRPO) significantly enhances the reasoning performance of Large Language Models (LLMs). However, this success heavily relies on expensive external verifiers or human rules. Such dependency not only leads to significant computational costs and training latency, but also yields sparse rewards that hinder optimization efficiency. To address these challenges, we propose Latent-GRPO, a framework that derives intrinsic rewards directly from latent space geometry. Crucially, our empirical analysis reveals a compelling geometric property: terminal token representations of correct reasoning trajectories form dense clusters with high intra-class similarity, whereas incorrect trajectories remain scattered as outliers. In light of this discovery, we introduce the Iterative Robust Centroid Estimation (IRCE) algorithm, which generates dense, continuous rewards by mitigating magnitude fluctuations via spherical projection and estimating a robust ``truth centroid'' through iterative aggregation. Experimental results on multiple datasets show that our method maintains model performance while achieving a training speedup of over 2x compared to baselines. Furthermore, extensive results demonstrate strong generalization ability and robustness. The code will be released soon. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_08427 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering Zhang, Nonghai Ma, Weitao Ma, Zhanyu Xu, Jun Gao, Jiuchong Hao, Jinghua He, Renqing Xu, Jingwen Computation and Language Machine Learning Group Relative Policy Optimization (GRPO) significantly enhances the reasoning performance of Large Language Models (LLMs). However, this success heavily relies on expensive external verifiers or human rules. Such dependency not only leads to significant computational costs and training latency, but also yields sparse rewards that hinder optimization efficiency. To address these challenges, we propose Latent-GRPO, a framework that derives intrinsic rewards directly from latent space geometry. Crucially, our empirical analysis reveals a compelling geometric property: terminal token representations of correct reasoning trajectories form dense clusters with high intra-class similarity, whereas incorrect trajectories remain scattered as outliers. In light of this discovery, we introduce the Iterative Robust Centroid Estimation (IRCE) algorithm, which generates dense, continuous rewards by mitigating magnitude fluctuations via spherical projection and estimating a robust ``truth centroid'' through iterative aggregation. Experimental results on multiple datasets show that our method maintains model performance while achieving a training speedup of over 2x compared to baselines. Furthermore, extensive results demonstrate strong generalization ability and robustness. The code will be released soon. |
| title | Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2601.08427 |