Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.22368 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910677803728896 |
|---|---|
| author | Spangher, Lucas Li, Tianle Arnold, William F. Masiewicki, Nick Dotiwalla, Xerxes Parusmathi, Rama Grabowski, Peter Ie, Eugene Gruhl, Dan |
| author_facet | Spangher, Lucas Li, Tianle Arnold, William F. Masiewicki, Nick Dotiwalla, Xerxes Parusmathi, Rama Grabowski, Peter Ie, Eugene Gruhl, Dan |
| contents | There exists an extremely wide array of LLM benchmarking tasks, whereas oftentimes a single number is the most actionable for decision-making, especially by non-experts. No such aggregation schema exists that is not Elo-based, which could be costly or time-consuming. Here we propose a method to aggregate performance across a general space of benchmarks, nicknamed Project "MPG," dubbed Model Performance and Goodness, additionally referencing a metric widely understood to be an important yet inaccurate and crude measure of car performance. Here, we create two numbers: a "Goodness" number (answer accuracy) and a "Fastness" number (cost or QPS). We compare models against each other and present a ranking according to our general metric as well as subdomains. We find significant agreement between the raw Pearson correlation of our scores and those of Chatbot Arena, even improving on the correlation of the MMLU leaderboard to Chatbot Arena. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_22368 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Project MPG: towards a generalized performance benchmark for LLM capabilities Spangher, Lucas Li, Tianle Arnold, William F. Masiewicki, Nick Dotiwalla, Xerxes Parusmathi, Rama Grabowski, Peter Ie, Eugene Gruhl, Dan Software Engineering Artificial Intelligence There exists an extremely wide array of LLM benchmarking tasks, whereas oftentimes a single number is the most actionable for decision-making, especially by non-experts. No such aggregation schema exists that is not Elo-based, which could be costly or time-consuming. Here we propose a method to aggregate performance across a general space of benchmarks, nicknamed Project "MPG," dubbed Model Performance and Goodness, additionally referencing a metric widely understood to be an important yet inaccurate and crude measure of car performance. Here, we create two numbers: a "Goodness" number (answer accuracy) and a "Fastness" number (cost or QPS). We compare models against each other and present a ranking according to our general metric as well as subdomains. We find significant agreement between the raw Pearson correlation of our scores and those of Chatbot Arena, even improving on the correlation of the MMLU leaderboard to Chatbot Arena. |
| title | Project MPG: towards a generalized performance benchmark for LLM capabilities |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2410.22368 |