SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929392385523712 |
|---|---|
| author | Dai, Josef Chen, Tianle Wang, Xuyao Yang, Ziran Chen, Taiye Ji, Jiaming Yang, Yaodong |
| author_facet | Dai, Josef Chen, Tianle Wang, Xuyao Yang, Ziran Chen, Taiye Ji, Jiaming Yang, Yaodong |
| contents | To mitigate the risk of harmful outputs from large vision models (LVMs), we introduce the SafeSora dataset to promote research on aligning text-to-video generation with human values. This dataset encompasses human preferences in text-to-video generation tasks along two primary dimensions: helpfulness and harmlessness. To capture in-depth human preferences and facilitate structured reasoning by crowdworkers, we subdivide helpfulness into 4 sub-dimensions and harmlessness into 12 sub-categories, serving as the basis for pilot annotations. The SafeSora dataset includes 14,711 unique prompts, 57,333 unique videos generated by 4 distinct LVMs, and 51,691 pairs of preference annotations labeled by humans. We further demonstrate the utility of the SafeSora dataset through several applications, including training the text-video moderation model and aligning LVMs with human preference by fine-tuning a prompt augmentation module or the diffusion model. These applications highlight its potential as the foundation for text-to-video alignment research, such as human preference modeling and the development and validation of alignment algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14477 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset Dai, Josef Chen, Tianle Wang, Xuyao Yang, Ziran Chen, Taiye Ji, Jiaming Yang, Yaodong Computer Vision and Pattern Recognition Artificial Intelligence Databases To mitigate the risk of harmful outputs from large vision models (LVMs), we introduce the SafeSora dataset to promote research on aligning text-to-video generation with human values. This dataset encompasses human preferences in text-to-video generation tasks along two primary dimensions: helpfulness and harmlessness. To capture in-depth human preferences and facilitate structured reasoning by crowdworkers, we subdivide helpfulness into 4 sub-dimensions and harmlessness into 12 sub-categories, serving as the basis for pilot annotations. The SafeSora dataset includes 14,711 unique prompts, 57,333 unique videos generated by 4 distinct LVMs, and 51,691 pairs of preference annotations labeled by humans. We further demonstrate the utility of the SafeSora dataset through several applications, including training the text-video moderation model and aligning LVMs with human preference by fine-tuning a prompt augmentation module or the diffusion model. These applications highlight its potential as the foundation for text-to-video alignment research, such as human preference modeling and the development and validation of alignment algorithms. |
| title | SafeSora: Towards Safety Alignment of Text2Video Generation via a Human Preference Dataset |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Databases |
| url | https://arxiv.org/abs/2406.14477 |