Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.24086 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908563737149440 |
|---|---|
| author | Gonzalez, Miguel Angel Alvarado Hernandez, Michelle Bruno Perez, Miguel Angel Peñaloza Orozco, Bruno Lopez Soto, Jesus Tadeo Cruz Malagon, Sandra |
| author_facet | Gonzalez, Miguel Angel Alvarado Hernandez, Michelle Bruno Perez, Miguel Angel Peñaloza Orozco, Bruno Lopez Soto, Jesus Tadeo Cruz Malagon, Sandra |
| contents | LLM leaderboards often rely on single stochastic runs, but how many repetitions are required for reliable conclusions remains unclear. We re-evaluate eight state-of-the-art models on the AI4Math Benchmark with three independent runs per setting. Using mixed-effects logistic regression, domain-level marginal means, rank-instability analysis, and run-to-run reliability, we assessed the value of additional repetitions. Our findings shows that Single-run leaderboards are brittle: 10/12 slices (83\%) invert at least one pairwise rank relative to the three-run majority, despite a zero sign-flip rate for pairwise significance and moderate overall interclass correlation. Averaging runs yields modest SE shrinkage ($\sim$5\% from one to three) but large ranking gains; two runs remove $\sim$83\% of single-run inversions. We provide cost-aware guidance for practitioners: treat evaluation as an experiment, report uncertainty, and use $\geq 2$ repetitions under stochastic decoding. These practices improve robustness while remaining feasible for small teams and help align model comparisons with real-world reliability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24086 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do Repetitions Matter? Strengthening Reliability in LLM Evaluations Gonzalez, Miguel Angel Alvarado Hernandez, Michelle Bruno Perez, Miguel Angel Peñaloza Orozco, Bruno Lopez Soto, Jesus Tadeo Cruz Malagon, Sandra Artificial Intelligence Computation and Language LLM leaderboards often rely on single stochastic runs, but how many repetitions are required for reliable conclusions remains unclear. We re-evaluate eight state-of-the-art models on the AI4Math Benchmark with three independent runs per setting. Using mixed-effects logistic regression, domain-level marginal means, rank-instability analysis, and run-to-run reliability, we assessed the value of additional repetitions. Our findings shows that Single-run leaderboards are brittle: 10/12 slices (83\%) invert at least one pairwise rank relative to the three-run majority, despite a zero sign-flip rate for pairwise significance and moderate overall interclass correlation. Averaging runs yields modest SE shrinkage ($\sim$5\% from one to three) but large ranking gains; two runs remove $\sim$83\% of single-run inversions. We provide cost-aware guidance for practitioners: treat evaluation as an experiment, report uncertainty, and use $\geq 2$ repetitions under stochastic decoding. These practices improve robustness while remaining feasible for small teams and help align model comparisons with real-world reliability. |
| title | Do Repetitions Matter? Strengthening Reliability in LLM Evaluations |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2509.24086 |