Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910271793004544 |
|---|---|
| author | Haupt, Andreas Hartenstein, Justin Reuel, Anka Kochenderfer, Mykel Koyejo, Sanmi |
| author_facet | Haupt, Andreas Hartenstein, Justin Reuel, Anka Kochenderfer, Mykel Koyejo, Sanmi |
| contents | AI benchmarks have well-documented limitations, with prior work examining contamination, saturation, and construct underspecification. Aggregation has received far less attention: benchmarks are typically summarized by uniformly averaging item-level scores, implicitly treating every test item as equally valuable. We model benchmarking as a multitask principal-agent game and show that the welfare loss from a benchmark is determined jointly by three item-level primitives: alignment with normative welfare priorities, marginal improvability, and performance variance. We translate the theory into an audit framework that ranks items along each of these three axes, and apply it to OLMES items using WORKBank for welfare, the EvoLM 4B suite for improvability, and the PolyPythias 410M panel for variance. The framework surfaces items that are Pareto-inferior within OLMES subject to a pro-worker welfare operationalization. All code is available at https://github.com/stair-lab/principal-agent-benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_30916 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation Haupt, Andreas Hartenstein, Justin Reuel, Anka Kochenderfer, Mykel Koyejo, Sanmi Machine Learning Computer Science and Game Theory Theoretical Economics AI benchmarks have well-documented limitations, with prior work examining contamination, saturation, and construct underspecification. Aggregation has received far less attention: benchmarks are typically summarized by uniformly averaging item-level scores, implicitly treating every test item as equally valuable. We model benchmarking as a multitask principal-agent game and show that the welfare loss from a benchmark is determined jointly by three item-level primitives: alignment with normative welfare priorities, marginal improvability, and performance variance. We translate the theory into an audit framework that ranks items along each of these three axes, and apply it to OLMES items using WORKBank for welfare, the EvoLM 4B suite for improvability, and the PolyPythias 410M panel for variance. The framework surfaces items that are Pareto-inferior within OLMES subject to a pro-worker welfare operationalization. All code is available at https://github.com/stair-lab/principal-agent-benchmarks. |
| title | Welfare, Improvability, and Variance: A Principal-Agent Approach to Optimal Benchmark Item Aggregation |
| topic | Machine Learning Computer Science and Game Theory Theoretical Economics |
| url | https://arxiv.org/abs/2605.30916 |