PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914621642768384 |
|---|---|
| author | Gao, Heyuan Tang, Bangxun Song, Yiren Fang, Guian He, Zijian Yang, Jie Shou, Mike Zheng |
| author_facet | Gao, Heyuan Tang, Bangxun Song, Yiren Fang, Guian He, Zijian Yang, Jie Shou, Mike Zheng |
| contents | We present PAI-Studio, a new reference-conditioned video synthesis task that addresses a long-standing challenge in cinematic background replacement: generating dynamic backgrounds aligned with foreground motion while preserving foreground identity, matching reference scene appearance, and achieving globally consistent illumination with realistic foreground relighting. Existing open-source systems and commercial APIs cannot simultaneously ensure motion-consistent background generation, high-fidelity foreground relighting and foreground identity preservation, often resulting in static backgrounds, inconsistent boundaries, and noticeable compositing artifacts. To bridge this gap, we build upon a Diffusion Transformer video backbone and reformulate the problem as an in-context conditional generation task. Through bidirectional attention, our model jointly captures foreground dynamics and background reference information within a unified architecture. We further construct a 30K-scale dataset sourced from high-quality films and online videos to support this task. Extensive evaluations demonstrate that our method significantly outperforms existing open-source and commercial API solutions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01399 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion Gao, Heyuan Tang, Bangxun Song, Yiren Fang, Guian He, Zijian Yang, Jie Shou, Mike Zheng Computer Vision and Pattern Recognition We present PAI-Studio, a new reference-conditioned video synthesis task that addresses a long-standing challenge in cinematic background replacement: generating dynamic backgrounds aligned with foreground motion while preserving foreground identity, matching reference scene appearance, and achieving globally consistent illumination with realistic foreground relighting. Existing open-source systems and commercial APIs cannot simultaneously ensure motion-consistent background generation, high-fidelity foreground relighting and foreground identity preservation, often resulting in static backgrounds, inconsistent boundaries, and noticeable compositing artifacts. To bridge this gap, we build upon a Diffusion Transformer video backbone and reformulate the problem as an in-context conditional generation task. Through bidirectional attention, our model jointly captures foreground dynamics and background reference information within a unified architecture. We further construct a 30K-scale dataset sourced from high-quality films and online videos to support this task. Extensive evaluations demonstrate that our method significantly outperforms existing open-source and commercial API solutions. |
| title | PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2606.01399 |