Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.10419 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908382273732608 |
|---|---|
| author | Nabati, Ofir Tennenholtz, Guy Hsu, ChihWei Ryu, Moonkyung Ramachandran, Deepak Chow, Yinlam Li, Xiang Boutilier, Craig |
| author_facet | Nabati, Ofir Tennenholtz, Guy Hsu, ChihWei Ryu, Moonkyung Ramachandran, Deepak Chow, Yinlam Li, Xiang Boutilier, Craig |
| contents | We address the problem of interactive text-to-image (T2I) generation, designing a reinforcement learning (RL) agent which iteratively improves a set of generated images for a user through a sequence of prompt expansions. Using human raters, we create a novel dataset of sequential preferences, which we leverage, together with large-scale open-source (non-sequential) datasets. We construct user-preference and user-choice models using an EM strategy and identify varying user preference types. We then leverage a large multimodal language model (LMM) and a value-based RL approach to suggest an adaptive and diverse slate of prompt expansions to the user. Our Preference Adaptive and Sequential Text-to-image Agent (PASTA) extends T2I models with adaptive multi-turn capabilities, fostering collaborative co-creation and addressing uncertainty or underspecification in a user's intent. We evaluate PASTA using human raters, showing significant improvement compared to baseline methods. We also open-source our sequential rater dataset and simulated user-rater interactions to support future research in user-centric multi-turn T2I systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_10419 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Preference Adaptive and Sequential Text-to-Image Generation Nabati, Ofir Tennenholtz, Guy Hsu, ChihWei Ryu, Moonkyung Ramachandran, Deepak Chow, Yinlam Li, Xiang Boutilier, Craig Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Systems and Control We address the problem of interactive text-to-image (T2I) generation, designing a reinforcement learning (RL) agent which iteratively improves a set of generated images for a user through a sequence of prompt expansions. Using human raters, we create a novel dataset of sequential preferences, which we leverage, together with large-scale open-source (non-sequential) datasets. We construct user-preference and user-choice models using an EM strategy and identify varying user preference types. We then leverage a large multimodal language model (LMM) and a value-based RL approach to suggest an adaptive and diverse slate of prompt expansions to the user. Our Preference Adaptive and Sequential Text-to-image Agent (PASTA) extends T2I models with adaptive multi-turn capabilities, fostering collaborative co-creation and addressing uncertainty or underspecification in a user's intent. We evaluate PASTA using human raters, showing significant improvement compared to baseline methods. We also open-source our sequential rater dataset and simulated user-rater interactions to support future research in user-centric multi-turn T2I systems. |
| title | Preference Adaptive and Sequential Text-to-Image Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2412.10419 |