Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.15542 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911520208715776 |
|---|---|
| author | Shi, Shaojie Shi, Zhengyu Zheng, Lingran Su, Xinyu Xie, Anna Lv, Bohao Xu, Rui Chen, Zijian Chen, Zhichao Liu, Guolei Zhang, Naifu Dong, Mingjian Quan, Zhuo Chen, Bohao Hao, Teqi Qi, Yuan Xu, Yinghui Wu, Libo |
| author_facet | Shi, Shaojie Shi, Zhengyu Zheng, Lingran Su, Xinyu Xie, Anna Lv, Bohao Xu, Rui Chen, Zijian Chen, Zhichao Liu, Guolei Zhang, Naifu Dong, Mingjian Quan, Zhuo Chen, Bohao Hao, Teqi Qi, Yuan Xu, Yinghui Wu, Libo |
| contents | Causal inference in social science relies on end-to-end, intervention-centered research-design reasoning grounded in real-world policy interventions, but current benchmarks fail to evaluate this capability of large language models (LLMs). We present InterveneBench, a benchmark designed to assess such reasoning in realistic social settings. Each instance in InterveneBench is derived from an empirical social science study and requires models to reason about policy interventions and identification assumptions without access to predefined causal graphs or structural equations. InterveneBench comprises 744 peer-reviewed studies across diverse policy domains. Experimental results show that state-of-the-art LLMs struggle under this setting. To address this limitation, we further propose a multi-agent framework, STRIDES. It achieves significant performance improvements over state-of-the-art reasoning models. Our code and data are available at https://github.com/Sii-yuning/STRIDES. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_15542 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systems Shi, Shaojie Shi, Zhengyu Zheng, Lingran Su, Xinyu Xie, Anna Lv, Bohao Xu, Rui Chen, Zijian Chen, Zhichao Liu, Guolei Zhang, Naifu Dong, Mingjian Quan, Zhuo Chen, Bohao Hao, Teqi Qi, Yuan Xu, Yinghui Wu, Libo Computers and Society Artificial Intelligence Causal inference in social science relies on end-to-end, intervention-centered research-design reasoning grounded in real-world policy interventions, but current benchmarks fail to evaluate this capability of large language models (LLMs). We present InterveneBench, a benchmark designed to assess such reasoning in realistic social settings. Each instance in InterveneBench is derived from an empirical social science study and requires models to reason about policy interventions and identification assumptions without access to predefined causal graphs or structural equations. InterveneBench comprises 744 peer-reviewed studies across diverse policy domains. Experimental results show that state-of-the-art LLMs struggle under this setting. To address this limitation, we further propose a multi-agent framework, STRIDES. It achieves significant performance improvements over state-of-the-art reasoning models. Our code and data are available at https://github.com/Sii-yuning/STRIDES. |
| title | InterveneBench: Benchmarking LLMs for Intervention Reasoning and Causal Study Design in Real Social Systems |
| topic | Computers and Society Artificial Intelligence |
| url | https://arxiv.org/abs/2603.15542 |