Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912417806548992 |
|---|---|
| author | Wallach, Hanna Desai, Meera Cooper, A. Feder Wang, Angelina Atalla, Chad Barocas, Solon Blodgett, Su Lin Chouldechova, Alexandra Corvi, Emily Dow, P. Alex Garcia-Gathright, Jean Olteanu, Alexandra Pangakis, Nicholas Reed, Stefanie Sheng, Emily Vann, Dan Vaughan, Jennifer Wortman Vogel, Matthew Washington, Hannah Jacobs, Abigail Z. |
| author_facet | Wallach, Hanna Desai, Meera Cooper, A. Feder Wang, Angelina Atalla, Chad Barocas, Solon Blodgett, Su Lin Chouldechova, Alexandra Corvi, Emily Dow, P. Alex Garcia-Gathright, Jean Olteanu, Alexandra Pangakis, Nicholas Reed, Stefanie Sheng, Emily Vann, Dan Vaughan, Jennifer Wortman Vogel, Matthew Washington, Hannah Jacobs, Abigail Z. |
| contents | The measurement tasks involved in evaluating generative AI (GenAI) systems lack sufficient scientific rigor, leading to what has been described as "a tangle of sloppy tests [and] apples-to-oranges comparisons" (Roose, 2024). In this position paper, we argue that the ML community would benefit from learning from and drawing on the social sciences when developing and using measurement instruments for evaluating GenAI systems. Specifically, our position is that evaluating GenAI systems is a social science measurement challenge. We present a four-level framework, grounded in measurement theory from the social sciences, for measuring concepts related to the capabilities, behaviors, and impacts of GenAI systems. This framework has two important implications: First, it can broaden the expertise involved in evaluating GenAI systems by enabling stakeholders with different perspectives to participate in conceptual debates. Second, it brings rigor to both conceptual and operational debates by offering a set of lenses for interrogating validity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_00561 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge Wallach, Hanna Desai, Meera Cooper, A. Feder Wang, Angelina Atalla, Chad Barocas, Solon Blodgett, Su Lin Chouldechova, Alexandra Corvi, Emily Dow, P. Alex Garcia-Gathright, Jean Olteanu, Alexandra Pangakis, Nicholas Reed, Stefanie Sheng, Emily Vann, Dan Vaughan, Jennifer Wortman Vogel, Matthew Washington, Hannah Jacobs, Abigail Z. Computers and Society The measurement tasks involved in evaluating generative AI (GenAI) systems lack sufficient scientific rigor, leading to what has been described as "a tangle of sloppy tests [and] apples-to-oranges comparisons" (Roose, 2024). In this position paper, we argue that the ML community would benefit from learning from and drawing on the social sciences when developing and using measurement instruments for evaluating GenAI systems. Specifically, our position is that evaluating GenAI systems is a social science measurement challenge. We present a four-level framework, grounded in measurement theory from the social sciences, for measuring concepts related to the capabilities, behaviors, and impacts of GenAI systems. This framework has two important implications: First, it can broaden the expertise involved in evaluating GenAI systems by enabling stakeholders with different perspectives to participate in conceptual debates. Second, it brings rigor to both conceptual and operational debates by offering a set of lenses for interrogating validity. |
| title | Position: Evaluating Generative AI Systems Is a Social Science Measurement Challenge |
| topic | Computers and Society |
| url | https://arxiv.org/abs/2502.00561 |