Tokenizing Motion: A Generative Approach for Scene Dynamics Compression
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912756517568512 |
|---|---|
| author | Yin, Shanzhi Zhang, Zihan Chen, Bolin Wang, Shiqi Ye, Yan |
| author_facet | Yin, Shanzhi Zhang, Zihan Chen, Bolin Wang, Shiqi Ye, Yan |
| contents | This paper proposes a novel generative video compression framework that leverages motion pattern priors, derived from subtle dynamics in common scenes (e.g., swaying flowers or a boat drifting on water), rather than relying on video content priors (e.g., talking faces or human bodies). These compact motion priors enable a new approach to ultra-low bitrate communication while achieving high-quality reconstruction across diverse scene contents. At the encoder side, motion priors can be streamlined into compact representations via a dense-to-sparse transformation. At the decoder side, these priors facilitate the reconstruction of scene dynamics using an advanced flow-driven diffusion model. Experimental results illustrate that the proposed method can achieve superior rate-distortion-performance and outperform the state-of-the-art conventional-video codec Enhanced Compression Model (ECM) on-scene dynamics sequences. The project page can be found at-https://github.com/xyzysz/GNVDC. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_09768 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Tokenizing Motion: A Generative Approach for Scene Dynamics Compression Yin, Shanzhi Zhang, Zihan Chen, Bolin Wang, Shiqi Ye, Yan Computer Vision and Pattern Recognition Image and Video Processing I.4; I.5 This paper proposes a novel generative video compression framework that leverages motion pattern priors, derived from subtle dynamics in common scenes (e.g., swaying flowers or a boat drifting on water), rather than relying on video content priors (e.g., talking faces or human bodies). These compact motion priors enable a new approach to ultra-low bitrate communication while achieving high-quality reconstruction across diverse scene contents. At the encoder side, motion priors can be streamlined into compact representations via a dense-to-sparse transformation. At the decoder side, these priors facilitate the reconstruction of scene dynamics using an advanced flow-driven diffusion model. Experimental results illustrate that the proposed method can achieve superior rate-distortion-performance and outperform the state-of-the-art conventional-video codec Enhanced Compression Model (ECM) on-scene dynamics sequences. The project page can be found at-https://github.com/xyzysz/GNVDC. |
| title | Tokenizing Motion: A Generative Approach for Scene Dynamics Compression |
| topic | Computer Vision and Pattern Recognition Image and Video Processing I.4; I.5 |
| url | https://arxiv.org/abs/2410.09768 |