Animated Stickers: Bringing Stickers to Life with Video Diffusion
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929238178791424 |
|---|---|
| author | Yan, David Zhang, Winnie Zhang, Luxin Kalia, Anmol Wang, Dingkang Ramchandani, Ankit Liu, Miao Pumarola, Albert Schoenfeld, Edgar Blanchard, Elliot Narni, Krishna Luo, Yaqiao Chen, Lawrence Pang, Guan Thabet, Ali Vajda, Peter Bearman, Amy Yu, Licheng |
| author_facet | Yan, David Zhang, Winnie Zhang, Luxin Kalia, Anmol Wang, Dingkang Ramchandani, Ankit Liu, Miao Pumarola, Albert Schoenfeld, Edgar Blanchard, Elliot Narni, Krishna Luo, Yaqiao Chen, Lawrence Pang, Guan Thabet, Ali Vajda, Peter Bearman, Amy Yu, Licheng |
| contents | We introduce animated stickers, a video diffusion model which generates an animation conditioned on a text prompt and static sticker image. Our model is built on top of the state-of-the-art Emu text-to-image model, with the addition of temporal layers to model motion. Due to the domain gap, i.e. differences in visual and motion style, a model which performed well on generating natural videos can no longer generate vivid videos when applied to stickers. To bridge this gap, we employ a two-stage finetuning pipeline: first with weakly in-domain data, followed by human-in-the-loop (HITL) strategy which we term ensemble-of-teachers. It distills the best qualities of multiple teachers into a smaller student model. We show that this strategy allows us to specifically target improvements to motion quality while maintaining the style from the static image. With inference optimizations, our model is able to generate an eight-frame video with high-quality, interesting, and relevant motion in under one second. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_06088 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Animated Stickers: Bringing Stickers to Life with Video Diffusion Yan, David Zhang, Winnie Zhang, Luxin Kalia, Anmol Wang, Dingkang Ramchandani, Ankit Liu, Miao Pumarola, Albert Schoenfeld, Edgar Blanchard, Elliot Narni, Krishna Luo, Yaqiao Chen, Lawrence Pang, Guan Thabet, Ali Vajda, Peter Bearman, Amy Yu, Licheng Computer Vision and Pattern Recognition We introduce animated stickers, a video diffusion model which generates an animation conditioned on a text prompt and static sticker image. Our model is built on top of the state-of-the-art Emu text-to-image model, with the addition of temporal layers to model motion. Due to the domain gap, i.e. differences in visual and motion style, a model which performed well on generating natural videos can no longer generate vivid videos when applied to stickers. To bridge this gap, we employ a two-stage finetuning pipeline: first with weakly in-domain data, followed by human-in-the-loop (HITL) strategy which we term ensemble-of-teachers. It distills the best qualities of multiple teachers into a smaller student model. We show that this strategy allows us to specifically target improvements to motion quality while maintaining the style from the static image. With inference optimizations, our model is able to generate an eight-frame video with high-quality, interesting, and relevant motion in under one second. |
| title | Animated Stickers: Bringing Stickers to Life with Video Diffusion |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.06088 |