Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909352290418688 |
|---|---|
| author | Collins, Katherine M. Kim, Najoung Bitton, Yonatan Rieser, Verena Omidshafiei, Shayegan Hu, Yushi Chen, Sherol Dutta, Senjuti Chang, Minsuk Lee, Kimin Liang, Youwei Evans, Georgina Singla, Sahil Li, Gang Weller, Adrian He, Junfeng Ramachandran, Deepak Dvijotham, Krishnamurthy Dj |
| author_facet | Collins, Katherine M. Kim, Najoung Bitton, Yonatan Rieser, Verena Omidshafiei, Shayegan Hu, Yushi Chen, Sherol Dutta, Senjuti Chang, Minsuk Lee, Kimin Liang, Youwei Evans, Georgina Singla, Sahil Li, Gang Weller, Adrian He, Junfeng Ramachandran, Deepak Dvijotham, Krishnamurthy Dj |
| contents | Human feedback plays a critical role in learning and refining reward models for text-to-image generation, but the optimal form the feedback should take for learning an accurate reward function has not been conclusively established. This paper investigates the effectiveness of fine-grained feedback which captures nuanced distinctions in image quality and prompt-alignment, compared to traditional coarse-grained feedback (for example, thumbs up/down or ranking between a set of options). While fine-grained feedback holds promise, particularly for systems catering to diverse societal preferences, we show that demonstrating its superiority to coarse-grained feedback is not automatic. Through experiments on real and synthetic preference data, we surface the complexities of building effective models due to the interplay of model choice, feedback type, and the alignment between human judgment and computational interpretation. We identify key challenges in eliciting and utilizing fine-grained feedback, prompting a reassessment of its assumed benefits and practicality. Our findings -- e.g., that fine-grained feedback can lead to worse models for a fixed budget, in some settings; however, in controlled settings with known attributes, fine grained rewards can indeed be more helpful -- call for careful consideration of feedback attributes and potentially beckon novel modeling approaches to appropriately unlock the potential value of fine-grained feedback in-the-wild. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16807 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation Collins, Katherine M. Kim, Najoung Bitton, Yonatan Rieser, Verena Omidshafiei, Shayegan Hu, Yushi Chen, Sherol Dutta, Senjuti Chang, Minsuk Lee, Kimin Liang, Youwei Evans, Georgina Singla, Sahil Li, Gang Weller, Adrian He, Junfeng Ramachandran, Deepak Dvijotham, Krishnamurthy Dj Machine Learning Computation and Language Computer Vision and Pattern Recognition Human feedback plays a critical role in learning and refining reward models for text-to-image generation, but the optimal form the feedback should take for learning an accurate reward function has not been conclusively established. This paper investigates the effectiveness of fine-grained feedback which captures nuanced distinctions in image quality and prompt-alignment, compared to traditional coarse-grained feedback (for example, thumbs up/down or ranking between a set of options). While fine-grained feedback holds promise, particularly for systems catering to diverse societal preferences, we show that demonstrating its superiority to coarse-grained feedback is not automatic. Through experiments on real and synthetic preference data, we surface the complexities of building effective models due to the interplay of model choice, feedback type, and the alignment between human judgment and computational interpretation. We identify key challenges in eliciting and utilizing fine-grained feedback, prompting a reassessment of its assumed benefits and practicality. Our findings -- e.g., that fine-grained feedback can lead to worse models for a fixed budget, in some settings; however, in controlled settings with known attributes, fine grained rewards can indeed be more helpful -- call for careful consideration of feedback attributes and potentially beckon novel modeling approaches to appropriately unlock the potential value of fine-grained feedback in-the-wild. |
| title | Beyond Thumbs Up/Down: Untangling Challenges of Fine-Grained Feedback for Text-to-Image Generation |
| topic | Machine Learning Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.16807 |