Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.03185 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914124831653888 |
|---|---|
| author | Zhao, Wanjia Ma, Qinwei Shi, Jingzhe Wu, Shirley Han, Jiaqi Xiao, Yijia Chen, Si-Yuan Luo, Xiao Schmidt, Ludwig Zou, James |
| author_facet | Zhao, Wanjia Ma, Qinwei Shi, Jingzhe Wu, Shirley Han, Jiaqi Xiao, Yijia Chen, Si-Yuan Luo, Xiao Schmidt, Ludwig Zou, James |
| contents | Benchmarks for competition-style reasoning have advanced evaluation in mathematics and programming, yet physics remains comparatively explored. Most existing physics benchmarks evaluate only final answers, which fail to capture reasoning processes, while recent stepwise methods rely on heuristic LLM-as-judge scoring or restrictive linear assumptions, limiting reliability and diagnostic validity. We introduce PRISM-Physics, a process-level evaluation framework and benchmark for complex physics reasoning problems. Solutions are represented as directed acyclic graphs (DAGs) of formulas, explicitly encoding causal dependencies among intermediate steps to enable fine-grained, interpretable, and theoretically grounded scoring. We prove the optimality of the DAG representation and the corresponding scoring policy. Combining with a fully rule-based method for symbolic formula equivalence matching that we developed, we ensure consistent validation across diverse formulations without heuristic judgments. Results show that our evaluation framework is more aligned with human experts' scoring. Experiments on state-of-the-art LLMs reveal persistent reasoning failures in physics, while step-level scoring offers both diagnostic insight and rich signals for later training. By combining structural rigor, theoretical guarantees, and symbolic validation, PRISM-Physics provides a principled foundation for advancing process-level evaluation and guiding the development of models with deeper scientific reasoning capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_03185 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning Zhao, Wanjia Ma, Qinwei Shi, Jingzhe Wu, Shirley Han, Jiaqi Xiao, Yijia Chen, Si-Yuan Luo, Xiao Schmidt, Ludwig Zou, James Machine Learning Benchmarks for competition-style reasoning have advanced evaluation in mathematics and programming, yet physics remains comparatively explored. Most existing physics benchmarks evaluate only final answers, which fail to capture reasoning processes, while recent stepwise methods rely on heuristic LLM-as-judge scoring or restrictive linear assumptions, limiting reliability and diagnostic validity. We introduce PRISM-Physics, a process-level evaluation framework and benchmark for complex physics reasoning problems. Solutions are represented as directed acyclic graphs (DAGs) of formulas, explicitly encoding causal dependencies among intermediate steps to enable fine-grained, interpretable, and theoretically grounded scoring. We prove the optimality of the DAG representation and the corresponding scoring policy. Combining with a fully rule-based method for symbolic formula equivalence matching that we developed, we ensure consistent validation across diverse formulations without heuristic judgments. Results show that our evaluation framework is more aligned with human experts' scoring. Experiments on state-of-the-art LLMs reveal persistent reasoning failures in physics, while step-level scoring offers both diagnostic insight and rich signals for later training. By combining structural rigor, theoretical guarantees, and symbolic validation, PRISM-Physics provides a principled foundation for advancing process-level evaluation and guiding the development of models with deeper scientific reasoning capabilities. |
| title | PRISM-Physics: Causal DAG-Based Process Evaluation for Physics Reasoning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.03185 |