Preference Adaptive and Sequential Text-to-Image Generation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908382273732608 |
|---|---|
| author | Nabati, Ofir Tennenholtz, Guy Hsu, ChihWei Ryu, Moonkyung Ramachandran, Deepak Chow, Yinlam Li, Xiang Boutilier, Craig |
| author_facet | Nabati, Ofir Tennenholtz, Guy Hsu, ChihWei Ryu, Moonkyung Ramachandran, Deepak Chow, Yinlam Li, Xiang Boutilier, Craig |
| contents | We address the problem of interactive text-to-image (T2I) generation, designing a reinforcement learning (RL) agent which iteratively improves a set of generated images for a user through a sequence of prompt expansions. Using human raters, we create a novel dataset of sequential preferences, which we leverage, together with large-scale open-source (non-sequential) datasets. We construct user-preference and user-choice models using an EM strategy and identify varying user preference types. We then leverage a large multimodal language model (LMM) and a value-based RL approach to suggest an adaptive and diverse slate of prompt expansions to the user. Our Preference Adaptive and Sequential Text-to-image Agent (PASTA) extends T2I models with adaptive multi-turn capabilities, fostering collaborative co-creation and addressing uncertainty or underspecification in a user's intent. We evaluate PASTA using human raters, showing significant improvement compared to baseline methods. We also open-source our sequential rater dataset and simulated user-rater interactions to support future research in user-centric multi-turn T2I systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_10419 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Preference Adaptive and Sequential Text-to-Image Generation Nabati, Ofir Tennenholtz, Guy Hsu, ChihWei Ryu, Moonkyung Ramachandran, Deepak Chow, Yinlam Li, Xiang Boutilier, Craig Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Systems and Control We address the problem of interactive text-to-image (T2I) generation, designing a reinforcement learning (RL) agent which iteratively improves a set of generated images for a user through a sequence of prompt expansions. Using human raters, we create a novel dataset of sequential preferences, which we leverage, together with large-scale open-source (non-sequential) datasets. We construct user-preference and user-choice models using an EM strategy and identify varying user preference types. We then leverage a large multimodal language model (LMM) and a value-based RL approach to suggest an adaptive and diverse slate of prompt expansions to the user. Our Preference Adaptive and Sequential Text-to-image Agent (PASTA) extends T2I models with adaptive multi-turn capabilities, fostering collaborative co-creation and addressing uncertainty or underspecification in a user's intent. We evaluate PASTA using human raters, showing significant improvement compared to baseline methods. We also open-source our sequential rater dataset and simulated user-rater interactions to support future research in user-centric multi-turn T2I systems. |
| title | Preference Adaptive and Sequential Text-to-Image Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2412.10419 |