ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911725853343744 |
|---|---|
| author | Li, Kenan Jin, Qirui Zhu, Liao Huang, Xiaosong Wu, Yijia Zhang, Yikai Zhang, Xin Jin, Zijian Huang, Yufan Nallipogu, Elsie Zhang, Chaoyun Kang, Yu Rajmohan, Saravan Lin, Qingwei Lee, Wenke Zhang, Dongmei |
| author_facet | Li, Kenan Jin, Qirui Zhu, Liao Huang, Xiaosong Wu, Yijia Zhang, Yikai Zhang, Xin Jin, Zijian Huang, Yufan Nallipogu, Elsie Zhang, Chaoyun Kang, Yu Rajmohan, Saravan Lin, Qingwei Lee, Wenke Zhang, Dongmei |
| contents | Recent advances in language model (LM) agents have significantly improved automated software engineering (SWE). Prior work has proposed various agentic workflows and training strategies as well as analyzed failure modes of agentic systems on SWE tasks, focusing on several contextual information signals: Reproduction Test, Regression Test, Edit Location, Execution Context, and API Usage. However, the individual contribution of each signal to overall success remains underexplored, particularly their ideal contribution when intermediate information is perfectly obtained. To address this gap, we introduce Oracle-SWE, a unified method to isolate and extract oracle information signals from SWE benchmarks and quantify the impact of each signal on agent performance. To further validate the pattern, we evaluate the performance gain of signals extracted by strong LMs when provided to a base agent, approximating real-world task-resolution settings. These evaluations aim to guide research prioritization for autonomous coding systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_07789 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents Li, Kenan Jin, Qirui Zhu, Liao Huang, Xiaosong Wu, Yijia Zhang, Yikai Zhang, Xin Jin, Zijian Huang, Yufan Nallipogu, Elsie Zhang, Chaoyun Kang, Yu Rajmohan, Saravan Lin, Qingwei Lee, Wenke Zhang, Dongmei Multiagent Systems Computation and Language Software Engineering I.2.7; I.2.5 Recent advances in language model (LM) agents have significantly improved automated software engineering (SWE). Prior work has proposed various agentic workflows and training strategies as well as analyzed failure modes of agentic systems on SWE tasks, focusing on several contextual information signals: Reproduction Test, Regression Test, Edit Location, Execution Context, and API Usage. However, the individual contribution of each signal to overall success remains underexplored, particularly their ideal contribution when intermediate information is perfectly obtained. To address this gap, we introduce Oracle-SWE, a unified method to isolate and extract oracle information signals from SWE benchmarks and quantify the impact of each signal on agent performance. To further validate the pattern, we evaluate the performance gain of signals extracted by strong LMs when provided to a base agent, approximating real-world task-resolution settings. These evaluations aim to guide research prioritization for autonomous coding systems. |
| title | ORACLE-SWE: Quantifying the Contribution of Oracle Information Signals on SWE Agents |
| topic | Multiagent Systems Computation and Language Software Engineering I.2.7; I.2.5 |
| url | https://arxiv.org/abs/2604.07789 |