PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914113199800320 |
|---|---|
| author | Chen, Yimeng Piȩkos, Piotr Ostaszewski, Mateusz Laakom, Firas Schmidhuber, Jürgen |
| author_facet | Chen, Yimeng Piȩkos, Piotr Ostaszewski, Mateusz Laakom, Firas Schmidhuber, Jürgen |
| contents | Evaluating the scientific discovery capabilities of large language model based agents, particularly how they cope with varying environmental complexity and utilize prior knowledge, requires specialized benchmarks currently lacking in the landscape. To address this gap, we introduce \textsc{PhysGym}, a novel benchmark suite and simulation platform for rigorously assessing LLM-based scientific reasoning in interactive physics environments. \textsc{PhysGym}'s primary contribution lies in its sophisticated control over the level of prior knowledge provided to the agent. This allows researchers to dissect agent performance along axes including the complexity of the problem and the prior knowledge levels. The benchmark comprises a suite of interactive simulations, where agents must actively probe environments, gather data sequentially under constraints and formulate hypotheses about underlying physical laws. \textsc{PhysGym} provides standardized evaluation protocols and metrics for assessing hypothesis accuracy and model fidelity. We demonstrate the benchmark's utility by presenting results from baseline LLMs, showcasing its ability to differentiate capabilities based on varying priors and task complexity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_15550 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors Chen, Yimeng Piȩkos, Piotr Ostaszewski, Mateusz Laakom, Firas Schmidhuber, Jürgen Machine Learning Artificial Intelligence Physics and Society Evaluating the scientific discovery capabilities of large language model based agents, particularly how they cope with varying environmental complexity and utilize prior knowledge, requires specialized benchmarks currently lacking in the landscape. To address this gap, we introduce \textsc{PhysGym}, a novel benchmark suite and simulation platform for rigorously assessing LLM-based scientific reasoning in interactive physics environments. \textsc{PhysGym}'s primary contribution lies in its sophisticated control over the level of prior knowledge provided to the agent. This allows researchers to dissect agent performance along axes including the complexity of the problem and the prior knowledge levels. The benchmark comprises a suite of interactive simulations, where agents must actively probe environments, gather data sequentially under constraints and formulate hypotheses about underlying physical laws. \textsc{PhysGym} provides standardized evaluation protocols and metrics for assessing hypothesis accuracy and model fidelity. We demonstrate the benchmark's utility by presenting results from baseline LLMs, showcasing its ability to differentiate capabilities based on varying priors and task complexity. |
| title | PhysGym: Benchmarking LLMs in Interactive Physics Discovery with Controlled Priors |
| topic | Machine Learning Artificial Intelligence Physics and Society |
| url | https://arxiv.org/abs/2507.15550 |