Evaluating the Social Impact of Generative AI Systems in Systems and Society
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908302972026880 |
|---|---|
| author | Solaiman, Irene Talat, Zeerak Agnew, William Ahmad, Lama Baker, Dylan Blodgett, Su Lin Chen, Canyu Daumé III, Hal Dodge, Jesse Duan, Isabella Evans, Ellie Friedrich, Felix Ghosh, Avijit Gohar, Usman Hooker, Sara Jernite, Yacine Kalluri, Ria Lusoli, Alberto Leidinger, Alina Lin, Michelle Lin, Xiuzhu Luccioni, Sasha Mickel, Jennifer Mitchell, Margaret Newman, Jessica Ovalle, Anaelia Png, Marie-Therese Singh, Shubham Strait, Andrew Struppek, Lukas Subramonian, Arjun |
| author_facet | Solaiman, Irene Talat, Zeerak Agnew, William Ahmad, Lama Baker, Dylan Blodgett, Su Lin Chen, Canyu Daumé III, Hal Dodge, Jesse Duan, Isabella Evans, Ellie Friedrich, Felix Ghosh, Avijit Gohar, Usman Hooker, Sara Jernite, Yacine Kalluri, Ria Lusoli, Alberto Leidinger, Alina Lin, Michelle Lin, Xiuzhu Luccioni, Sasha Mickel, Jennifer Mitchell, Margaret Newman, Jessica Ovalle, Anaelia Png, Marie-Therese Singh, Shubham Strait, Andrew Struppek, Lukas Subramonian, Arjun |
| contents | Generative AI systems across modalities, ranging from text (including code), image, audio, and video, have broad social impacts, but there is no official standard for means of evaluating those impacts or for which impacts should be evaluated. In this paper, we present a guide that moves toward a standard approach in evaluating a base generative AI system for any modality in two overarching categories: what can be evaluated in a base system independent of context and what can be evaluated in a societal context. Importantly, this refers to base systems that have no predetermined application or deployment context, including a model itself, as well as system components, such as training data. Our framework for a base system defines seven categories of social impact: bias, stereotypes, and representational harms; cultural values and sensitive content; disparate performance; privacy and data protection; financial costs; environmental costs; and data and content moderation labor costs. Suggested methods for evaluation apply to listed generative modalities and analyses of the limitations of existing evaluations serve as a starting point for necessary investment in future evaluations. We offer five overarching categories for what can be evaluated in a broader societal context, each with its own subcategories: trustworthiness and autonomy; inequality, marginalization, and violence; concentration of authority; labor and creativity; and ecosystem and environment. Each subcategory includes recommendations for mitigating harm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_05949 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Evaluating the Social Impact of Generative AI Systems in Systems and Society Solaiman, Irene Talat, Zeerak Agnew, William Ahmad, Lama Baker, Dylan Blodgett, Su Lin Chen, Canyu Daumé III, Hal Dodge, Jesse Duan, Isabella Evans, Ellie Friedrich, Felix Ghosh, Avijit Gohar, Usman Hooker, Sara Jernite, Yacine Kalluri, Ria Lusoli, Alberto Leidinger, Alina Lin, Michelle Lin, Xiuzhu Luccioni, Sasha Mickel, Jennifer Mitchell, Margaret Newman, Jessica Ovalle, Anaelia Png, Marie-Therese Singh, Shubham Strait, Andrew Struppek, Lukas Subramonian, Arjun Computers and Society Artificial Intelligence Generative AI systems across modalities, ranging from text (including code), image, audio, and video, have broad social impacts, but there is no official standard for means of evaluating those impacts or for which impacts should be evaluated. In this paper, we present a guide that moves toward a standard approach in evaluating a base generative AI system for any modality in two overarching categories: what can be evaluated in a base system independent of context and what can be evaluated in a societal context. Importantly, this refers to base systems that have no predetermined application or deployment context, including a model itself, as well as system components, such as training data. Our framework for a base system defines seven categories of social impact: bias, stereotypes, and representational harms; cultural values and sensitive content; disparate performance; privacy and data protection; financial costs; environmental costs; and data and content moderation labor costs. Suggested methods for evaluation apply to listed generative modalities and analyses of the limitations of existing evaluations serve as a starting point for necessary investment in future evaluations. We offer five overarching categories for what can be evaluated in a broader societal context, each with its own subcategories: trustworthiness and autonomy; inequality, marginalization, and violence; concentration of authority; labor and creativity; and ecosystem and environment. Each subcategory includes recommendations for mitigating harm. |
| title | Evaluating the Social Impact of Generative AI Systems in Systems and Society |
| topic | Computers and Society Artificial Intelligence |
| url | https://arxiv.org/abs/2306.05949 |