Auto-Discovery-Bench: Diagnosing Structured State Tracking in Oracle-Guided Discovery
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914614970679296 |
|---|---|
| author | Chen, Tingting Lin, Beibei Anumasa, Srinivas Shah, Vedant Yuan, Zifeng Zou, Qiran Goyal, Anirudh Liu, Dianbo |
| author_facet | Chen, Tingting Lin, Beibei Anumasa, Srinivas Shah, Vedant Yuan, Zifeng Zou, Qiran Goyal, Anirudh Liu, Dianbo |
| contents | Interactive discovery requires agents to maintain and update structured beliefs over many rounds of feedback. Before evaluating agents in noisy, open-ended scientific environments, it is useful to isolate this prerequisite capability under controlled conditions. We introduce Auto-Discovery-Bench, a deterministic oracle-guided diagnostic benchmark in which agents recover hidden structures through repeated hypothesis--intervention--feedback cycles. The benchmark instantiates three controlled discovery abstractions: directed graph discovery, undirected relational discovery, and symbolic equation discovery. Across models, performance degrades as the number of variables, trajectory length, and distractors increase. A separate trajectory-tracking diagnostic shows that many failures persist even when intervention selection and hypothesis generation are removed, suggesting that limitations in maintaining and integrating long-range structured information are an important bottleneck for oracle-guided discovery. Auto-Discovery-Bench is not intended to replace realistic discovery environments; rather, it provides a reproducible, low-confound diagnostic testbed for isolating a prerequisite capability for interactive scientific agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15224 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Auto-Discovery-Bench: Diagnosing Structured State Tracking in Oracle-Guided Discovery Chen, Tingting Lin, Beibei Anumasa, Srinivas Shah, Vedant Yuan, Zifeng Zou, Qiran Goyal, Anirudh Liu, Dianbo Machine Learning Artificial Intelligence Interactive discovery requires agents to maintain and update structured beliefs over many rounds of feedback. Before evaluating agents in noisy, open-ended scientific environments, it is useful to isolate this prerequisite capability under controlled conditions. We introduce Auto-Discovery-Bench, a deterministic oracle-guided diagnostic benchmark in which agents recover hidden structures through repeated hypothesis--intervention--feedback cycles. The benchmark instantiates three controlled discovery abstractions: directed graph discovery, undirected relational discovery, and symbolic equation discovery. Across models, performance degrades as the number of variables, trajectory length, and distractors increase. A separate trajectory-tracking diagnostic shows that many failures persist even when intervention selection and hypothesis generation are removed, suggesting that limitations in maintaining and integrating long-range structured information are an important bottleneck for oracle-guided discovery. Auto-Discovery-Bench is not intended to replace realistic discovery environments; rather, it provides a reproducible, low-confound diagnostic testbed for isolating a prerequisite capability for interactive scientific agents. |
| title | Auto-Discovery-Bench: Diagnosing Structured State Tracking in Oracle-Guided Discovery |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.15224 |