Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.08140 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909644970000384 |
|---|---|
| author | Li, Yifei Moussa, Hanane Nour Chen, Ziru Chen, Shijie Yu, Botao Xue, Mingyi Burns, Benjamin Chiu, Tzu-Yao Dey, Vishal Lu, Zitong Wei, Chen Zhang, Qianheng Zhang, Tianyu Gao, Song Huang, Xuhui Ning, Xia Ahmed, Nesreen K. Payani, Ali Sun, Huan |
| author_facet | Li, Yifei Moussa, Hanane Nour Chen, Ziru Chen, Shijie Yu, Botao Xue, Mingyi Burns, Benjamin Chiu, Tzu-Yao Dey, Vishal Lu, Zitong Wei, Chen Zhang, Qianheng Zhang, Tianyu Gao, Song Huang, Xuhui Ning, Xia Ahmed, Nesreen K. Payani, Ali Sun, Huan |
| contents | Despite long-standing efforts in accelerating scientific discovery with AI, building AI co-scientists remains challenging due to limited high-quality data for training and evaluation. To tackle this data scarcity issue, we present AutoSDT, an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. AutoSDT leverages the coding capabilities and parametric knowledge of LLMs to search for diverse sources, select ecologically valid tasks, and synthesize accurate task instructions and code solutions. Using our pipeline, we construct AutoSDT-5K, a dataset of 5,404 coding tasks for data-driven discovery that covers four scientific disciplines and 756 unique Python packages. To the best of our knowledge, AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery. Expert feedback on a subset of 256 tasks shows the effectiveness of AutoSDT: 93% of the collected tasks are ecologically valid, and 92.2% of the synthesized programs are functionally correct. Trained on AutoSDT-5K, the Qwen2.5-Coder-Instruct LLM series, dubbed AutoSDT-Coder, show substantial improvement on two challenging data-driven discovery benchmarks, ScienceAgentBench and DiscoveryBench. Most notably, AutoSDT-Coder-32B reaches the same level of performance as GPT-4o on ScienceAgentBench with a success rate of 7.8%, doubling the performance of its base model. On DiscoveryBench, it lifts the hypothesis matching score to 8.1, bringing a 17.4% relative improvement and closing the gap between open-weight models and GPT-4o. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08140 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists Li, Yifei Moussa, Hanane Nour Chen, Ziru Chen, Shijie Yu, Botao Xue, Mingyi Burns, Benjamin Chiu, Tzu-Yao Dey, Vishal Lu, Zitong Wei, Chen Zhang, Qianheng Zhang, Tianyu Gao, Song Huang, Xuhui Ning, Xia Ahmed, Nesreen K. Payani, Ali Sun, Huan Machine Learning Computation and Language Despite long-standing efforts in accelerating scientific discovery with AI, building AI co-scientists remains challenging due to limited high-quality data for training and evaluation. To tackle this data scarcity issue, we present AutoSDT, an automatic pipeline that collects high-quality coding tasks in real-world data-driven discovery workflows. AutoSDT leverages the coding capabilities and parametric knowledge of LLMs to search for diverse sources, select ecologically valid tasks, and synthesize accurate task instructions and code solutions. Using our pipeline, we construct AutoSDT-5K, a dataset of 5,404 coding tasks for data-driven discovery that covers four scientific disciplines and 756 unique Python packages. To the best of our knowledge, AutoSDT-5K is the only automatically collected and the largest open dataset for data-driven scientific discovery. Expert feedback on a subset of 256 tasks shows the effectiveness of AutoSDT: 93% of the collected tasks are ecologically valid, and 92.2% of the synthesized programs are functionally correct. Trained on AutoSDT-5K, the Qwen2.5-Coder-Instruct LLM series, dubbed AutoSDT-Coder, show substantial improvement on two challenging data-driven discovery benchmarks, ScienceAgentBench and DiscoveryBench. Most notably, AutoSDT-Coder-32B reaches the same level of performance as GPT-4o on ScienceAgentBench with a success rate of 7.8%, doubling the performance of its base model. On DiscoveryBench, it lifts the hypothesis matching score to 8.1, bringing a 17.4% relative improvement and closing the gap between open-weight models and GPT-4o. |
| title | AutoSDT: Scaling Data-Driven Discovery Tasks Toward Open Co-Scientists |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2506.08140 |