Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913295776088064 |
|---|---|
| author | Kim, Kyuyoung Jeong, Jongheon An, Minyong Ghavamzadeh, Mohammad Dvijotham, Krishnamurthy Shin, Jinwoo Lee, Kimin |
| author_facet | Kim, Kyuyoung Jeong, Jongheon An, Minyong Ghavamzadeh, Mohammad Dvijotham, Krishnamurthy Shin, Jinwoo Lee, Kimin |
| contents | Fine-tuning text-to-image models with reward functions trained on human feedback data has proven effective for aligning model behavior with human intent. However, excessive optimization with such reward models, which serve as mere proxy objectives, can compromise the performance of fine-tuned models, a phenomenon known as reward overoptimization. To investigate this issue in depth, we introduce the Text-Image Alignment Assessment (TIA2) benchmark, which comprises a diverse collection of text prompts, images, and human annotations. Our evaluation of several state-of-the-art reward models on this benchmark reveals their frequent misalignment with human assessment. We empirically demonstrate that overoptimization occurs notably when a poorly aligned reward model is used as the fine-tuning objective. To address this, we propose TextNorm, a simple method that enhances alignment based on a measure of reward model confidence estimated across a set of semantically contrastive text prompts. We demonstrate that incorporating the confidence-calibrated rewards in fine-tuning effectively reduces overoptimization, resulting in twice as many wins in human evaluation for text-image alignment compared against the baseline reward models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_01863 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models Kim, Kyuyoung Jeong, Jongheon An, Minyong Ghavamzadeh, Mohammad Dvijotham, Krishnamurthy Shin, Jinwoo Lee, Kimin Machine Learning Artificial Intelligence Fine-tuning text-to-image models with reward functions trained on human feedback data has proven effective for aligning model behavior with human intent. However, excessive optimization with such reward models, which serve as mere proxy objectives, can compromise the performance of fine-tuned models, a phenomenon known as reward overoptimization. To investigate this issue in depth, we introduce the Text-Image Alignment Assessment (TIA2) benchmark, which comprises a diverse collection of text prompts, images, and human annotations. Our evaluation of several state-of-the-art reward models on this benchmark reveals their frequent misalignment with human assessment. We empirically demonstrate that overoptimization occurs notably when a poorly aligned reward model is used as the fine-tuning objective. To address this, we propose TextNorm, a simple method that enhances alignment based on a measure of reward model confidence estimated across a set of semantically contrastive text prompts. We demonstrate that incorporating the confidence-calibrated rewards in fine-tuning effectively reduces overoptimization, resulting in twice as many wins in human evaluation for text-image alignment compared against the baseline reward models. |
| title | Confidence-aware Reward Optimization for Fine-tuning Text-to-Image Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2404.01863 |