BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913871528198144 |
|---|---|
| author | Sokol, Anna Daly, Elizabeth Hind, Michael Piorkowski, David Zhang, Xiangliang Moniz, Nuno Chawla, Nitesh |
| author_facet | Sokol, Anna Daly, Elizabeth Hind, Michael Piorkowski, David Zhang, Xiangliang Moniz, Nuno Chawla, Nitesh |
| contents | Large language models (LLMs) are powerful tools capable of handling diverse tasks. Comparing and selecting appropriate LLMs for specific tasks requires systematic evaluation methods, as models exhibit varying capabilities across different domains. However, finding suitable benchmarks is difficult given the many available options. This complexity not only increases the risk of benchmark misuse and misinterpretation but also demands substantial effort from LLM users, seeking the most suitable benchmarks for their specific needs. To address these issues, we introduce \texttt{BenchmarkCards}, an intuitive and validated documentation framework that standardizes critical benchmark attributes such as objectives, methodologies, data sources, and limitations. Through user studies involving benchmark creators and users, we show that \texttt{BenchmarkCards} can simplify benchmark selection and enhance transparency, facilitating informed decision-making in evaluating LLMs. Data & Code: https://github.com/SokolAnn/BenchmarkCards |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12974 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks Sokol, Anna Daly, Elizabeth Hind, Michael Piorkowski, David Zhang, Xiangliang Moniz, Nuno Chawla, Nitesh Computation and Language Large language models (LLMs) are powerful tools capable of handling diverse tasks. Comparing and selecting appropriate LLMs for specific tasks requires systematic evaluation methods, as models exhibit varying capabilities across different domains. However, finding suitable benchmarks is difficult given the many available options. This complexity not only increases the risk of benchmark misuse and misinterpretation but also demands substantial effort from LLM users, seeking the most suitable benchmarks for their specific needs. To address these issues, we introduce \texttt{BenchmarkCards}, an intuitive and validated documentation framework that standardizes critical benchmark attributes such as objectives, methodologies, data sources, and limitations. Through user studies involving benchmark creators and users, we show that \texttt{BenchmarkCards} can simplify benchmark selection and enhance transparency, facilitating informed decision-making in evaluating LLMs. Data & Code: https://github.com/SokolAnn/BenchmarkCards |
| title | BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.12974 |