Auto-BenchmarkCard: Automated Synthesis of Benchmark Documentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914192574906368 |
|---|---|
| author | Hofmann, Aris Vejsbjerg, Inge Salwala, Dhaval Daly, Elizabeth M. |
| author_facet | Hofmann, Aris Vejsbjerg, Inge Salwala, Dhaval Daly, Elizabeth M. |
| contents | We present Auto-BenchmarkCard, a workflow for generating validated descriptions of AI benchmarks. Benchmark documentation is often incomplete or inconsistent, making it difficult to interpret and compare benchmarks across tasks or domains. Auto-BenchmarkCard addresses this gap by combining multi-agent data extraction from heterogeneous sources (e.g., Hugging Face, Unitxt, academic papers) with LLM-driven synthesis. A validation phase evaluates factual accuracy through atomic entailment scoring using the FactReasoner tool. This workflow has the potential to promote transparency, comparability, and reusability in AI benchmark reporting, enabling researchers and practitioners to better navigate and evaluate benchmark choices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_09577 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Auto-BenchmarkCard: Automated Synthesis of Benchmark Documentation Hofmann, Aris Vejsbjerg, Inge Salwala, Dhaval Daly, Elizabeth M. Human-Computer Interaction Artificial Intelligence We present Auto-BenchmarkCard, a workflow for generating validated descriptions of AI benchmarks. Benchmark documentation is often incomplete or inconsistent, making it difficult to interpret and compare benchmarks across tasks or domains. Auto-BenchmarkCard addresses this gap by combining multi-agent data extraction from heterogeneous sources (e.g., Hugging Face, Unitxt, academic papers) with LLM-driven synthesis. A validation phase evaluates factual accuracy through atomic entailment scoring using the FactReasoner tool. This workflow has the potential to promote transparency, comparability, and reusability in AI benchmark reporting, enabling researchers and practitioners to better navigate and evaluate benchmark choices. |
| title | Auto-BenchmarkCard: Automated Synthesis of Benchmark Documentation |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2512.09577 |