Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.10541 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913175278977024 |
|---|---|
| author | Chen, Zihan Zhang, Yiming Zhou, Hengguang Ding, Zenghui Sun, Yining Hsieh, Cho-Jui |
| author_facet | Chen, Zihan Zhang, Yiming Zhou, Hengguang Ding, Zenghui Sun, Yining Hsieh, Cho-Jui |
| contents | Current benchmarks are inadequate for evaluating progress in reinforcement learning (RL) for large language models (LLMs).Despite recent benchmark gains reported for RL, we find that training on these benchmarks' training sets achieves nearly the same performance as training directly on the test sets, suggesting that the benchmarks cannot reliably separate further progress.To study this phenomenon, we introduce a diagnostic suite and the Oracle Performance Gap (OPG) metric that quantifies the performance difference between training on the train split versus the test split of a benchmark. We further analyze this phenomenon with stress tests and find that, despite strong benchmark scores, existing RL methods struggle to generalize across distribution shifts, varying levels of difficulty, and counterfactual scenarios: shortcomings that current benchmarks fail to reveal.We conclude that current benchmarks are insufficient for evaluating generalization and propose three core principles for designing more faithful benchmarks: sufficient difficulty, balanced evaluation, and distributional robustness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10541 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods? Chen, Zihan Zhang, Yiming Zhou, Hengguang Ding, Zenghui Sun, Yining Hsieh, Cho-Jui Machine Learning Artificial Intelligence Current benchmarks are inadequate for evaluating progress in reinforcement learning (RL) for large language models (LLMs).Despite recent benchmark gains reported for RL, we find that training on these benchmarks' training sets achieves nearly the same performance as training directly on the test sets, suggesting that the benchmarks cannot reliably separate further progress.To study this phenomenon, we introduce a diagnostic suite and the Oracle Performance Gap (OPG) metric that quantifies the performance difference between training on the train split versus the test split of a benchmark. We further analyze this phenomenon with stress tests and find that, despite strong benchmark scores, existing RL methods struggle to generalize across distribution shifts, varying levels of difficulty, and counterfactual scenarios: shortcomings that current benchmarks fail to reveal.We conclude that current benchmarks are insufficient for evaluating generalization and propose three core principles for designing more faithful benchmarks: sufficient difficulty, balanced evaluation, and distributional robustness. |
| title | Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods? |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.10541 |