Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2511.05613 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911735345053696 |
|---|---|
| author | Reuel, Anka Ghosh, Avijit Chim, Jenny Tran, Andrew Long, Yanan Mickel, Jennifer Gohar, Usman Yadav, Srishti Ammanamanchi, Pawan Sasanka Allaham, Mowafak Rahmani, Hossein A. Akhtar, Mubashara Friedrich, Felix Scholz, Robert Riegler, Michael Alexander Batzner, Jan Habba, Eliya Saxena, Arushi Kornilova, Anastassia Wei, Kevin Soni, Prajna Mathew, Yohan Klyman, Kevin Sania, Jeba Sahoo, Subramanyam Bruvik, Olivia Beyer Sadeghi, Pouya Goswami, Sujata Wang, Angelina Jernite, Yacine Talat, Zeerak Biderman, Stella Kochenderfer, Mykel Koyejo, Sanmi Solaiman, Irene |
| author_facet | Reuel, Anka Ghosh, Avijit Chim, Jenny Tran, Andrew Long, Yanan Mickel, Jennifer Gohar, Usman Yadav, Srishti Ammanamanchi, Pawan Sasanka Allaham, Mowafak Rahmani, Hossein A. Akhtar, Mubashara Friedrich, Felix Scholz, Robert Riegler, Michael Alexander Batzner, Jan Habba, Eliya Saxena, Arushi Kornilova, Anastassia Wei, Kevin Soni, Prajna Mathew, Yohan Klyman, Kevin Sania, Jeba Sahoo, Subramanyam Bruvik, Olivia Beyer Sadeghi, Pouya Goswami, Sujata Wang, Angelina Jernite, Yacine Talat, Zeerak Biderman, Stella Kochenderfer, Mykel Koyejo, Sanmi Solaiman, Irene |
| contents | Foundation models are increasingly central to high-stakes AI systems, and governance frameworks now depend on evaluations to assess their risks and capabilities. Although general capability evaluations are widespread, social impact assessments covering bias, fairness, privacy, environmental costs, and labor remain uneven. To characterize this landscape, we conduct the first comprehensive analysis of social impact evaluation reporting, examining 186 first-party release reports and 248 third-party evaluation sources, supplemented by developer interviews. We find a stark division of labor: first-party reporting is sparse, often superficial, and declining in areas like environmental impact and bias, while third-party evaluators provide broader, more rigorous coverage of bias, harmful content, and performance disparities. However, only developers can authoritatively report on data provenance, content moderation labor, costs, and infrastructure, yet interviews reveal these disclosures are deprioritized unless tied to product adoption or compliance. Current practices leave major gaps in assessing societal impacts, underscoring the need for policies that mandate developer transparency, strengthen independent evaluation ecosystems, and create shared infrastructure for aggregating third-party evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_05613 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations Reuel, Anka Ghosh, Avijit Chim, Jenny Tran, Andrew Long, Yanan Mickel, Jennifer Gohar, Usman Yadav, Srishti Ammanamanchi, Pawan Sasanka Allaham, Mowafak Rahmani, Hossein A. Akhtar, Mubashara Friedrich, Felix Scholz, Robert Riegler, Michael Alexander Batzner, Jan Habba, Eliya Saxena, Arushi Kornilova, Anastassia Wei, Kevin Soni, Prajna Mathew, Yohan Klyman, Kevin Sania, Jeba Sahoo, Subramanyam Bruvik, Olivia Beyer Sadeghi, Pouya Goswami, Sujata Wang, Angelina Jernite, Yacine Talat, Zeerak Biderman, Stella Kochenderfer, Mykel Koyejo, Sanmi Solaiman, Irene Computers and Society Artificial Intelligence Machine Learning Foundation models are increasingly central to high-stakes AI systems, and governance frameworks now depend on evaluations to assess their risks and capabilities. Although general capability evaluations are widespread, social impact assessments covering bias, fairness, privacy, environmental costs, and labor remain uneven. To characterize this landscape, we conduct the first comprehensive analysis of social impact evaluation reporting, examining 186 first-party release reports and 248 third-party evaluation sources, supplemented by developer interviews. We find a stark division of labor: first-party reporting is sparse, often superficial, and declining in areas like environmental impact and bias, while third-party evaluators provide broader, more rigorous coverage of bias, harmful content, and performance disparities. However, only developers can authoritatively report on data provenance, content moderation labor, costs, and infrastructure, yet interviews reveal these disclosures are deprioritized unless tied to product adoption or compliance. Current practices leave major gaps in assessing societal impacts, underscoring the need for policies that mandate developer transparency, strengthen independent evaluation ecosystems, and create shared infrastructure for aggregating third-party evaluations. |
| title | Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations |
| topic | Computers and Society Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2511.05613 |