Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.05576 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910358988390400 |
|---|---|
| author | Dutta, Senjuti Chen, Sherol Mak, Sunny Ahmad, Amnah Collins, Katherine Butryna, Alena Ramachandran, Deepak Dvijotham, Krishnamurthy Pavlick, Ellie Rajakumar, Ravi |
| author_facet | Dutta, Senjuti Chen, Sherol Mak, Sunny Ahmad, Amnah Collins, Katherine Butryna, Alena Ramachandran, Deepak Dvijotham, Krishnamurthy Pavlick, Ellie Rajakumar, Ravi |
| contents | Image generation models are poised to become ubiquitous in a range of applications. These models are often fine-tuned and evaluated using human quality judgments that assume a universal standard, failing to consider the subjectivity of such tasks. To investigate how to quantify subjectivity, and the scale of its impact, we measure how assessments differ among human annotators across different use cases. Simulating the effects of ordinarily latent elements of annotators subjectivity, we contrive a set of motivations (t-shirt graphics, presentation visuals, and phone background images) to contextualize a set of crowdsourcing tasks. Our results show that human evaluations of images vary within individual contexts and across combinations of contexts. Three key factors affecting this subjectivity are image appearance, image alignment with text, and representation of objects mentioned in the text. Our study highlights the importance of taking individual users and contexts into account, both when building and evaluating generative models |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_05576 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Understanding Subjectivity through the Lens of Motivational Context in Model-Generated Image Satisfaction Dutta, Senjuti Chen, Sherol Mak, Sunny Ahmad, Amnah Collins, Katherine Butryna, Alena Ramachandran, Deepak Dvijotham, Krishnamurthy Pavlick, Ellie Rajakumar, Ravi Human-Computer Interaction Artificial Intelligence Image generation models are poised to become ubiquitous in a range of applications. These models are often fine-tuned and evaluated using human quality judgments that assume a universal standard, failing to consider the subjectivity of such tasks. To investigate how to quantify subjectivity, and the scale of its impact, we measure how assessments differ among human annotators across different use cases. Simulating the effects of ordinarily latent elements of annotators subjectivity, we contrive a set of motivations (t-shirt graphics, presentation visuals, and phone background images) to contextualize a set of crowdsourcing tasks. Our results show that human evaluations of images vary within individual contexts and across combinations of contexts. Three key factors affecting this subjectivity are image appearance, image alignment with text, and representation of objects mentioned in the text. Our study highlights the importance of taking individual users and contexts into account, both when building and evaluating generative models |
| title | Understanding Subjectivity through the Lens of Motivational Context in Model-Generated Image Satisfaction |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2403.05576 |