Bayesian Prediction-Powered Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913345742831616 |
|---|---|
| author | Hofer, R. Alex Maynez, Joshua Dhingra, Bhuwan Fisch, Adam Globerson, Amir Cohen, William W. |
| author_facet | Hofer, R. Alex Maynez, Joshua Dhingra, Bhuwan Fisch, Adam Globerson, Amir Cohen, William W. |
| contents | Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data. Specifically, PPI methods provide tighter confidence intervals by combining small amounts of human-labeled data with larger amounts of data labeled by a reasonably accurate, but potentially biased, automatic system. We propose a framework for PPI based on Bayesian inference that allows researchers to develop new task-appropriate PPI methods easily. Exploiting the ease with which we can design new metrics, we propose improved PPI methods for several importantcases, such as autoraters that give discrete responses (e.g., prompted LLM ``judges'') and autoraters with scores that have a non-linear relationship to human scores. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_06034 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Bayesian Prediction-Powered Inference Hofer, R. Alex Maynez, Joshua Dhingra, Bhuwan Fisch, Adam Globerson, Amir Cohen, William W. Machine Learning Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data. Specifically, PPI methods provide tighter confidence intervals by combining small amounts of human-labeled data with larger amounts of data labeled by a reasonably accurate, but potentially biased, automatic system. We propose a framework for PPI based on Bayesian inference that allows researchers to develop new task-appropriate PPI methods easily. Exploiting the ease with which we can design new metrics, we propose improved PPI methods for several importantcases, such as autoraters that give discrete responses (e.g., prompted LLM ``judges'') and autoraters with scores that have a non-linear relationship to human scores. |
| title | Bayesian Prediction-Powered Inference |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2405.06034 |