Handling Missing Responses under Cluster Dependence with Applications to Language Model Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912668118417408 |
|---|---|
| author | Zeng, Zhenghao Arbour, David Feller, Avi Dasgupta, Ishita Sinha, Atanu R Kennedy, Edward H. |
| author_facet | Zeng, Zhenghao Arbour, David Feller, Avi Dasgupta, Ishita Sinha, Atanu R Kennedy, Edward H. |
| contents | Human annotations play a crucial role in evaluating the performance of GenAI models. Two common challenges in practice, however, are missing annotations (the response variable of interest) and cluster dependence among human-AI interactions (e.g., questions asked by the same user may be highly correlated). Reliable inference must address both these issues to achieve unbiased estimation and appropriately quantify uncertainty when estimating average scores from human annotations. In this paper, we analyze the doubly robust estimator, a widely used method in missing data analysis and causal inference, applied to this setting and establish novel theoretical properties under cluster dependence. We further illustrate our findings through simulations and a real-world conversation quality dataset. Our theoretical and empirical results underscore the importance of incorporating cluster dependence in missing response problems to perform valid statistical inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_20928 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Handling Missing Responses under Cluster Dependence with Applications to Language Model Evaluation Zeng, Zhenghao Arbour, David Feller, Avi Dasgupta, Ishita Sinha, Atanu R Kennedy, Edward H. Methodology Human annotations play a crucial role in evaluating the performance of GenAI models. Two common challenges in practice, however, are missing annotations (the response variable of interest) and cluster dependence among human-AI interactions (e.g., questions asked by the same user may be highly correlated). Reliable inference must address both these issues to achieve unbiased estimation and appropriately quantify uncertainty when estimating average scores from human annotations. In this paper, we analyze the doubly robust estimator, a widely used method in missing data analysis and causal inference, applied to this setting and establish novel theoretical properties under cluster dependence. We further illustrate our findings through simulations and a real-world conversation quality dataset. Our theoretical and empirical results underscore the importance of incorporating cluster dependence in missing response problems to perform valid statistical inference. |
| title | Handling Missing Responses under Cluster Dependence with Applications to Language Model Evaluation |
| topic | Methodology |
| url | https://arxiv.org/abs/2510.20928 |