RSPO: Risk-Seeking Policy Optimization for Pass@k and Max@k Metrics in Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909718580035584 |
|---|---|
| author | Zhang, Kaichen Gao, Shenghao Hong, Yuzhong Sun, Haipeng Bao, Junwei Jiang, Hongfei Song, Yang Dingqian, Hong Xiong, Hui |
| author_facet | Zhang, Kaichen Gao, Shenghao Hong, Yuzhong Sun, Haipeng Bao, Junwei Jiang, Hongfei Song, Yang Dingqian, Hong Xiong, Hui |
| contents | Current large language model post-training optimizes a risk-neutral objective that maximizes expected reward, yet evaluation relies heavily on risk-seeking metrics like Pass@k (at least one success in k trials) and Max@k (maximum reward across k responses). This mismatch in risk preferences can inevitably lead to suboptimal performance. To bridge this gap, we propose Risk-Seeking Policy Optimization (RSPO), a novel method that directly targets Pass@k and Max@k during training. A key challenge in optimizing these metrics is the "hitchhiking" problem: low-reward responses are inadvertently reinforced if they co-occur with a high-reward response within a sample of k generations, resulting in inefficient optimization. RSPO addresses this problem by leveraging the closed-form probability that a given response is the maximum among k samplings. Despite the complexity of nested gradients over multiple responses, RSPO produces efficient, unbiased gradient estimators for both metrics. We validate our approach with both rigorous theoretical analysis and comprehensive experimental results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_01174 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RSPO: Risk-Seeking Policy Optimization for Pass@k and Max@k Metrics in Large Language Models Zhang, Kaichen Gao, Shenghao Hong, Yuzhong Sun, Haipeng Bao, Junwei Jiang, Hongfei Song, Yang Dingqian, Hong Xiong, Hui Machine Learning Artificial Intelligence Current large language model post-training optimizes a risk-neutral objective that maximizes expected reward, yet evaluation relies heavily on risk-seeking metrics like Pass@k (at least one success in k trials) and Max@k (maximum reward across k responses). This mismatch in risk preferences can inevitably lead to suboptimal performance. To bridge this gap, we propose Risk-Seeking Policy Optimization (RSPO), a novel method that directly targets Pass@k and Max@k during training. A key challenge in optimizing these metrics is the "hitchhiking" problem: low-reward responses are inadvertently reinforced if they co-occur with a high-reward response within a sample of k generations, resulting in inefficient optimization. RSPO addresses this problem by leveraging the closed-form probability that a given response is the maximum among k samplings. Despite the complexity of nested gradients over multiple responses, RSPO produces efficient, unbiased gradient estimators for both metrics. We validate our approach with both rigorous theoretical analysis and comprehensive experimental results. |
| title | RSPO: Risk-Seeking Policy Optimization for Pass@k and Max@k Metrics in Large Language Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.01174 |