Multi-subject Open-set Personalization in Video Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915206064504832 |
|---|---|
| author | Chen, Tsai-Shien Siarohin, Aliaksandr Menapace, Willi Fang, Yuwei Lee, Kwot Sin Skorokhodov, Ivan Aberman, Kfir Zhu, Jun-Yan Yang, Ming-Hsuan Tulyakov, Sergey |
| author_facet | Chen, Tsai-Shien Siarohin, Aliaksandr Menapace, Willi Fang, Yuwei Lee, Kwot Sin Skorokhodov, Ivan Aberman, Kfir Zhu, Jun-Yan Yang, Ming-Hsuan Tulyakov, Sergey |
| contents | Video personalization methods allow us to synthesize videos with specific concepts such as people, pets, and places. However, existing methods often focus on limited domains, require time-consuming optimization per subject, or support only a single subject. We present Video Alchemist $-$ a video model with built-in multi-subject, open-set personalization capabilities for both foreground objects and background, eliminating the need for time-consuming test-time optimization. Our model is built on a new Diffusion Transformer module that fuses each conditional reference image and its corresponding subject-level text prompt with cross-attention layers. Developing such a large model presents two main challenges: dataset and evaluation. First, as paired datasets of reference images and videos are extremely hard to collect, we sample selected video frames as reference images and synthesize a clip of the target video. However, while models can easily denoise training videos given reference frames, they fail to generalize to new contexts. To mitigate this issue, we design a new automatic data construction pipeline with extensive image augmentations. Second, evaluating open-set video personalization is a challenge in itself. To address this, we introduce a personalization benchmark that focuses on accurate subject fidelity and supports diverse personalization scenarios. Finally, our extensive experiments show that our method significantly outperforms existing personalization methods in both quantitative and qualitative evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_06187 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multi-subject Open-set Personalization in Video Generation Chen, Tsai-Shien Siarohin, Aliaksandr Menapace, Willi Fang, Yuwei Lee, Kwot Sin Skorokhodov, Ivan Aberman, Kfir Zhu, Jun-Yan Yang, Ming-Hsuan Tulyakov, Sergey Computer Vision and Pattern Recognition Video personalization methods allow us to synthesize videos with specific concepts such as people, pets, and places. However, existing methods often focus on limited domains, require time-consuming optimization per subject, or support only a single subject. We present Video Alchemist $-$ a video model with built-in multi-subject, open-set personalization capabilities for both foreground objects and background, eliminating the need for time-consuming test-time optimization. Our model is built on a new Diffusion Transformer module that fuses each conditional reference image and its corresponding subject-level text prompt with cross-attention layers. Developing such a large model presents two main challenges: dataset and evaluation. First, as paired datasets of reference images and videos are extremely hard to collect, we sample selected video frames as reference images and synthesize a clip of the target video. However, while models can easily denoise training videos given reference frames, they fail to generalize to new contexts. To mitigate this issue, we design a new automatic data construction pipeline with extensive image augmentations. Second, evaluating open-set video personalization is a challenge in itself. To address this, we introduce a personalization benchmark that focuses on accurate subject fidelity and supports diverse personalization scenarios. Finally, our extensive experiments show that our method significantly outperforms existing personalization methods in both quantitative and qualitative evaluations. |
| title | Multi-subject Open-set Personalization in Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.06187 |