Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.01787 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916359347109888 |
|---|---|
| author | Hao, Susan Shelby, Renee Liu, Yuchi Srinivasan, Hansa Bhutani, Mukul Ayan, Burcu Karagol Poplin, Ryan Poddar, Shivani Laszlo, Sarah |
| author_facet | Hao, Susan Shelby, Renee Liu, Yuchi Srinivasan, Hansa Bhutani, Mukul Ayan, Burcu Karagol Poplin, Ryan Poddar, Shivani Laszlo, Sarah |
| contents | Text-to-image (T2I) models have emerged as a significant advancement in generative AI; however, there exist safety concerns regarding their potential to produce harmful image outputs even when users input seemingly safe prompts. This phenomenon, where T2I models generate harmful representations that were not explicit in the input prompt, poses a potentially greater risk than adversarial prompts, leaving users unintentionally exposed to harms. Our paper addresses this issue by formalizing a definition for this phenomenon which we term harm amplification. We further contribute to the field by developing a framework of methodologies to quantify harm amplification in which we consider the harm of the model output in the context of user input. We then empirically examine how to apply these different methodologies to simulate real-world deployment scenarios including a quantification of disparate impacts across genders resulting from harm amplification. Together, our work aims to offer researchers tools to comprehensively address safety challenges in T2I systems and contribute to the responsible deployment of generative AI models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01787 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Harm Amplification in Text-to-Image Models Hao, Susan Shelby, Renee Liu, Yuchi Srinivasan, Hansa Bhutani, Mukul Ayan, Burcu Karagol Poplin, Ryan Poddar, Shivani Laszlo, Sarah Computers and Society Artificial Intelligence Machine Learning Text-to-image (T2I) models have emerged as a significant advancement in generative AI; however, there exist safety concerns regarding their potential to produce harmful image outputs even when users input seemingly safe prompts. This phenomenon, where T2I models generate harmful representations that were not explicit in the input prompt, poses a potentially greater risk than adversarial prompts, leaving users unintentionally exposed to harms. Our paper addresses this issue by formalizing a definition for this phenomenon which we term harm amplification. We further contribute to the field by developing a framework of methodologies to quantify harm amplification in which we consider the harm of the model output in the context of user input. We then empirically examine how to apply these different methodologies to simulate real-world deployment scenarios including a quantification of disparate impacts across genders resulting from harm amplification. Together, our work aims to offer researchers tools to comprehensively address safety challenges in T2I systems and contribute to the responsible deployment of generative AI models. |
| title | Harm Amplification in Text-to-Image Models |
| topic | Computers and Society Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2402.01787 |