What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915944037613568 |
|---|---|
| author | Luo, Yujie Yu, Zhuoyun Wang, Xuehai Zhu, Yuqi Zhang, Ningyu Wei, Lanning Du, Lun Zheng, Da Chen, Huajun |
| author_facet | Luo, Yujie Yu, Zhuoyun Wang, Xuehai Zhu, Yuqi Zhang, Ningyu Wei, Lanning Du, Lun Zheng, Da Chen, Huajun |
| contents | Replicating AI research is a crucial yet challenging task for large language model (LLM) agents. Existing approaches often struggle to generate executable code, primarily due to insufficient background knowledge and the limitations of retrieval-augmented generation (RAG) methods, which fail to capture latent technical details hidden in referenced papers. Furthermore, previous approaches tend to overlook valuable implementation-level code signals and lack structured knowledge representations that support multi-granular retrieval and reuse. To overcome these challenges, we propose Executable Knowledge Graphs (xKG), a pluggable, paper-centric knowledge base that automatically integrates code snippets and technical insights extracted from scientific literature. When integrated into three agent frameworks with two different LLMs, xKG shows substantial performance gains (10.9% with o3-mini) on PaperBench, demonstrating its effectiveness as a general and extensible solution for automated AI research replication. Code is available at https://github.com/zjunlp/xKG. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_17795 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations Luo, Yujie Yu, Zhuoyun Wang, Xuehai Zhu, Yuqi Zhang, Ningyu Wei, Lanning Du, Lun Zheng, Da Chen, Huajun Computation and Language Artificial Intelligence Machine Learning Multiagent Systems Software Engineering Replicating AI research is a crucial yet challenging task for large language model (LLM) agents. Existing approaches often struggle to generate executable code, primarily due to insufficient background knowledge and the limitations of retrieval-augmented generation (RAG) methods, which fail to capture latent technical details hidden in referenced papers. Furthermore, previous approaches tend to overlook valuable implementation-level code signals and lack structured knowledge representations that support multi-granular retrieval and reuse. To overcome these challenges, we propose Executable Knowledge Graphs (xKG), a pluggable, paper-centric knowledge base that automatically integrates code snippets and technical insights extracted from scientific literature. When integrated into three agent frameworks with two different LLMs, xKG shows substantial performance gains (10.9% with o3-mini) on PaperBench, demonstrating its effectiveness as a general and extensible solution for automated AI research replication. Code is available at https://github.com/zjunlp/xKG. |
| title | What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations |
| topic | Computation and Language Artificial Intelligence Machine Learning Multiagent Systems Software Engineering |
| url | https://arxiv.org/abs/2510.17795 |