PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908400065970176 |
|---|---|
| author | Gong, Albert Stankevičiūtė, Kamilė Wan, Chao Kabra, Anmol Thesmar, Raphael Lee, Johann Klenke, Julius Gomes, Carla P. Weinberger, Kilian Q. |
| author_facet | Gong, Albert Stankevičiūtė, Kamilė Wan, Chao Kabra, Anmol Thesmar, Raphael Lee, Johann Klenke, Julius Gomes, Carla P. Weinberger, Kilian Q. |
| contents | High-quality benchmarks are essential for evaluating reasoning and retrieval capabilities of large language models (LLMs). However, curating datasets for this purpose is not a permanent solution as they are prone to data leakage and inflated performance results. To address these challenges, we propose PhantomWiki: a pipeline to generate unique, factually consistent document corpora with diverse question-answer pairs. Unlike prior work, PhantomWiki is neither a fixed dataset, nor is it based on any existing data. Instead, a new PhantomWiki instance is generated on demand for each evaluation. We vary the question difficulty and corpus size to disentangle reasoning and retrieval capabilities respectively, and find that PhantomWiki datasets are surprisingly challenging for frontier LLMs. Thus, we contribute a scalable and data leakage-resistant framework for disentangled evaluation of reasoning, retrieval, and tool-use abilities. Our code is available at https://github.com/kilian-group/phantom-wiki. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20377 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation Gong, Albert Stankevičiūtė, Kamilė Wan, Chao Kabra, Anmol Thesmar, Raphael Lee, Johann Klenke, Julius Gomes, Carla P. Weinberger, Kilian Q. Machine Learning Artificial Intelligence Computation and Language High-quality benchmarks are essential for evaluating reasoning and retrieval capabilities of large language models (LLMs). However, curating datasets for this purpose is not a permanent solution as they are prone to data leakage and inflated performance results. To address these challenges, we propose PhantomWiki: a pipeline to generate unique, factually consistent document corpora with diverse question-answer pairs. Unlike prior work, PhantomWiki is neither a fixed dataset, nor is it based on any existing data. Instead, a new PhantomWiki instance is generated on demand for each evaluation. We vary the question difficulty and corpus size to disentangle reasoning and retrieval capabilities respectively, and find that PhantomWiki datasets are surprisingly challenging for frontier LLMs. Thus, we contribute a scalable and data leakage-resistant framework for disentangled evaluation of reasoning, retrieval, and tool-use abilities. Our code is available at https://github.com/kilian-group/phantom-wiki. |
| title | PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2502.20377 |