Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909711884877824 |
|---|---|
| author | Ficek, Aleksander Majumdar, Somshubra Noroozi, Vahid Ginsburg, Boris |
| author_facet | Ficek, Aleksander Majumdar, Somshubra Noroozi, Vahid Ginsburg, Boris |
| contents | Synthetic verification techniques such as generating test cases and reward modelling are common ways to enhance the coding capabilities of large language models (LLM) beyond predefined tests. Additionally, code verification has recently found great success as a critical component in improving reasoning capability of LLMs via reinforcement learning. In this paper, we propose an approach which can transform existing coding benchmarks into scoring and ranking datasets to evaluate the effectiveness of synthetic verifiers. We also propose multiple metrics to measure different aspects of the synthetic verifiers with the proposed benchmarks. By employing the proposed approach, we release four new benchmarks (HE-R, HE-R+, MBPP-R, and MBPP-R+), and analyzed synthetic verification methods with standard, reasoning-based, and reward-based LLMs. Our experiments show that reasoning can significantly improve test case generation and that scaling the number of test cases enhances the verification accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13820 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning Ficek, Aleksander Majumdar, Somshubra Noroozi, Vahid Ginsburg, Boris Artificial Intelligence Computation and Language Machine Learning Software Engineering Synthetic verification techniques such as generating test cases and reward modelling are common ways to enhance the coding capabilities of large language models (LLM) beyond predefined tests. Additionally, code verification has recently found great success as a critical component in improving reasoning capability of LLMs via reinforcement learning. In this paper, we propose an approach which can transform existing coding benchmarks into scoring and ranking datasets to evaluate the effectiveness of synthetic verifiers. We also propose multiple metrics to measure different aspects of the synthetic verifiers with the proposed benchmarks. By employing the proposed approach, we release four new benchmarks (HE-R, HE-R+, MBPP-R, and MBPP-R+), and analyzed synthetic verification methods with standard, reasoning-based, and reward-based LLMs. Our experiments show that reasoning can significantly improve test case generation and that scaling the number of test cases enhances the verification accuracy. |
| title | Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning |
| topic | Artificial Intelligence Computation and Language Machine Learning Software Engineering |
| url | https://arxiv.org/abs/2502.13820 |