L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910908833333248 |
|---|---|
| author | Sun, Simeng Hsieh, Cheng-Ping Ladhak, Faisal Arakelyan, Erik Serano, Santiago Akle Ginsburg, Boris |
| author_facet | Sun, Simeng Hsieh, Cheng-Ping Ladhak, Faisal Arakelyan, Erik Serano, Santiago Akle Ginsburg, Boris |
| contents | Complex reasoning tasks often rely on the ability to consistently and accurately apply simple rules across incremental steps, a foundational capability which we term "level-0" reasoning. To systematically evaluate this capability, we introduce L0-Bench, a language model benchmark for testing procedural correctness -- the ability to generate correct reasoning processes, complementing existing benchmarks that primarily focus on outcome correctness. Given synthetic Python functions with simple operations, L0-Bench grades models on their ability to generate step-by-step, error-free execution traces. The synthetic nature of L0-Bench enables systematic and scalable generation of test programs along various axes (e.g., number of trace steps). We evaluate a diverse array of recent closed-source and open-weight models on a baseline test set. All models exhibit degradation as the number of target trace steps increases, while larger models and reasoning-enhanced models better maintain correctness over multiple steps. Additionally, we use L0-Bench to explore test-time scaling along three dimensions: input context length, number of solutions for majority voting, and inference steps. Our results suggest substantial room to improve "level-0" reasoning and potential directions to build more reliable reasoning systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_22832 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution Sun, Simeng Hsieh, Cheng-Ping Ladhak, Faisal Arakelyan, Erik Serano, Santiago Akle Ginsburg, Boris Programming Languages Computation and Language Complex reasoning tasks often rely on the ability to consistently and accurately apply simple rules across incremental steps, a foundational capability which we term "level-0" reasoning. To systematically evaluate this capability, we introduce L0-Bench, a language model benchmark for testing procedural correctness -- the ability to generate correct reasoning processes, complementing existing benchmarks that primarily focus on outcome correctness. Given synthetic Python functions with simple operations, L0-Bench grades models on their ability to generate step-by-step, error-free execution traces. The synthetic nature of L0-Bench enables systematic and scalable generation of test programs along various axes (e.g., number of trace steps). We evaluate a diverse array of recent closed-source and open-weight models on a baseline test set. All models exhibit degradation as the number of target trace steps increases, while larger models and reasoning-enhanced models better maintain correctness over multiple steps. Additionally, we use L0-Bench to explore test-time scaling along three dimensions: input context length, number of solutions for majority voting, and inference steps. Our results suggest substantial room to improve "level-0" reasoning and potential directions to build more reliable reasoning systems. |
| title | L0-Reasoning Bench: Evaluating Procedural Correctness in Language Models via Simple Program Execution |
| topic | Programming Languages Computation and Language |
| url | https://arxiv.org/abs/2503.22832 |