Auto-BenchmarkCard: Automated Synthesis of Benchmark Documentation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914192574906368 |
|---|---|
| author | Hofmann, Aris Vejsbjerg, Inge Salwala, Dhaval Daly, Elizabeth M. |
| author_facet | Hofmann, Aris Vejsbjerg, Inge Salwala, Dhaval Daly, Elizabeth M. |
| contents | We present Auto-BenchmarkCard, a workflow for generating validated descriptions of AI benchmarks. Benchmark documentation is often incomplete or inconsistent, making it difficult to interpret and compare benchmarks across tasks or domains. Auto-BenchmarkCard addresses this gap by combining multi-agent data extraction from heterogeneous sources (e.g., Hugging Face, Unitxt, academic papers) with LLM-driven synthesis. A validation phase evaluates factual accuracy through atomic entailment scoring using the FactReasoner tool. This workflow has the potential to promote transparency, comparability, and reusability in AI benchmark reporting, enabling researchers and practitioners to better navigate and evaluate benchmark choices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_09577 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Auto-BenchmarkCard: Automated Synthesis of Benchmark Documentation Hofmann, Aris Vejsbjerg, Inge Salwala, Dhaval Daly, Elizabeth M. Human-Computer Interaction Artificial Intelligence We present Auto-BenchmarkCard, a workflow for generating validated descriptions of AI benchmarks. Benchmark documentation is often incomplete or inconsistent, making it difficult to interpret and compare benchmarks across tasks or domains. Auto-BenchmarkCard addresses this gap by combining multi-agent data extraction from heterogeneous sources (e.g., Hugging Face, Unitxt, academic papers) with LLM-driven synthesis. A validation phase evaluates factual accuracy through atomic entailment scoring using the FactReasoner tool. This workflow has the potential to promote transparency, comparability, and reusability in AI benchmark reporting, enabling researchers and practitioners to better navigate and evaluate benchmark choices. |
| title | Auto-BenchmarkCard: Automated Synthesis of Benchmark Documentation |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2512.09577 |