Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2505.24098 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913866761371648 |
|---|---|
| author | He, Zhongmou Choi, Yee Man Zhang, Kexun Ji, Jiabao Zhou, Junting Xu, Dejia Bercovich, Ivan Zhang, Aidan Li, Lei |
| author_facet | He, Zhongmou Choi, Yee Man Zhang, Kexun Ji, Jiabao Zhou, Junting Xu, Dejia Bercovich, Ivan Zhang, Aidan Li, Lei |
| contents | Verifiers play a crucial role in large language model (LLM) reasoning, needed by post-training techniques such as reinforcement learning. However, reliable verifiers are hard to get for difficult coding problems, because a well-disguised wrong solution may only be detected by carefully human-written edge cases that are difficult to synthesize. To address this issue, we propose HARDTESTGEN, a pipeline for high-quality test synthesis using LLMs. With this pipeline, we curate a comprehensive competitive programming dataset HARDTESTS with 47k problems and synthetic high-quality tests. Compared with existing tests, HARDTESTGEN tests demonstrate precision that is 11.3 percentage points higher and recall that is 17.5 percentage points higher when evaluating LLM-generated code. For harder problems, the improvement in precision can be as large as 40 points. HARDTESTS also proves to be more effective for model training, measured by downstream code generation performance. We will open-source our dataset and synthesis pipeline at https://leililab.github.io/HardTests/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_24098 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HardTests: Synthesizing High-Quality Test Cases for LLM Coding He, Zhongmou Choi, Yee Man Zhang, Kexun Ji, Jiabao Zhou, Junting Xu, Dejia Bercovich, Ivan Zhang, Aidan Li, Lei Computation and Language Verifiers play a crucial role in large language model (LLM) reasoning, needed by post-training techniques such as reinforcement learning. However, reliable verifiers are hard to get for difficult coding problems, because a well-disguised wrong solution may only be detected by carefully human-written edge cases that are difficult to synthesize. To address this issue, we propose HARDTESTGEN, a pipeline for high-quality test synthesis using LLMs. With this pipeline, we curate a comprehensive competitive programming dataset HARDTESTS with 47k problems and synthetic high-quality tests. Compared with existing tests, HARDTESTGEN tests demonstrate precision that is 11.3 percentage points higher and recall that is 17.5 percentage points higher when evaluating LLM-generated code. For harder problems, the improvement in precision can be as large as 40 points. HARDTESTS also proves to be more effective for model training, measured by downstream code generation performance. We will open-source our dataset and synthesis pipeline at https://leililab.github.io/HardTests/. |
| title | HardTests: Synthesizing High-Quality Test Cases for LLM Coding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.24098 |