Pointwise Metrics Mislead: An Evaluation Protocol for Multimodal Inverse Problems
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916037785550848 |
|---|---|
| author | Baattrup, Mads H. Bach, Jörn Jeppe, Laurids Labe, Finn Grohsjean, Alexander Schwanenberger, Christian Stelldinger, Peer |
| author_facet | Baattrup, Mads H. Bach, Jörn Jeppe, Laurids Labe, Finn Grohsjean, Alexander Schwanenberger, Christian Stelldinger, Peer |
| contents | Evaluation in scientific reconstruction is dominated by pointwise metrics - RMSE, MAE, per-event resolution - under the implicit assumption that lower error means better reconstruction. We show that this assumption fails structurally for inverse problems with multimodal posteriors. By the law of total variance, point estimators trained to minimize MSE or MAE produce a marginal spectrum strictly narrower than the truth whenever the posterior has nonzero width. The resulting bias is independent of architecture, training, and dataset size, and it compresses precisely the spectral features - tails, modes, shapes - that downstream scientific measurements rely on. We propose a three-part evaluation protocol where each step targets a failure mode the others miss: per-event distributional accuracy via CRPS, population-level marginal accuracy via a spectrum-fidelity diagnostic, and uncertainty trustworthiness via coverage-based calibration. On a synthetic benchmark with an analytic posterior and on a realistic many-to-one inverse problem from particle physics, model rankings reverse between pointwise and distributional metrics, and calibration further separates architectures indistinguishable under CRPS. The evaluation protocol, not the model, determines the scientific conclusion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_22891 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Pointwise Metrics Mislead: An Evaluation Protocol for Multimodal Inverse Problems Baattrup, Mads H. Bach, Jörn Jeppe, Laurids Labe, Finn Grohsjean, Alexander Schwanenberger, Christian Stelldinger, Peer Machine Learning High Energy Physics - Experiment Evaluation in scientific reconstruction is dominated by pointwise metrics - RMSE, MAE, per-event resolution - under the implicit assumption that lower error means better reconstruction. We show that this assumption fails structurally for inverse problems with multimodal posteriors. By the law of total variance, point estimators trained to minimize MSE or MAE produce a marginal spectrum strictly narrower than the truth whenever the posterior has nonzero width. The resulting bias is independent of architecture, training, and dataset size, and it compresses precisely the spectral features - tails, modes, shapes - that downstream scientific measurements rely on. We propose a three-part evaluation protocol where each step targets a failure mode the others miss: per-event distributional accuracy via CRPS, population-level marginal accuracy via a spectrum-fidelity diagnostic, and uncertainty trustworthiness via coverage-based calibration. On a synthetic benchmark with an analytic posterior and on a realistic many-to-one inverse problem from particle physics, model rankings reverse between pointwise and distributional metrics, and calibration further separates architectures indistinguishable under CRPS. The evaluation protocol, not the model, determines the scientific conclusion. |
| title | Pointwise Metrics Mislead: An Evaluation Protocol for Multimodal Inverse Problems |
| topic | Machine Learning High Energy Physics - Experiment |
| url | https://arxiv.org/abs/2605.22891 |