Towards Online Real-Time Memory-based Video Inpainting Transformers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911811794632704 |
|---|---|
| author | Thiry, Guillaume Tang, Hao Timofte, Radu Van Gool, Luc |
| author_facet | Thiry, Guillaume Tang, Hao Timofte, Radu Van Gool, Luc |
| contents | Video inpainting tasks have seen significant improvements in recent years with the rise of deep neural networks and, in particular, vision transformers. Although these models show promising reconstruction quality and temporal consistency, they are still unsuitable for live videos, one of the last steps to make them completely convincing and usable. The main limitations are that these state-of-the-art models inpaint using the whole video (offline processing) and show an insufficient frame rate. In our approach, we propose a framework to adapt existing inpainting transformers to these constraints by memorizing and refining redundant computations while maintaining a decent inpainting quality. Using this framework with some of the most recent inpainting models, we show great online results with a consistent throughput above 20 frames per second. The code and pretrained models will be made available upon acceptance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_16161 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Online Real-Time Memory-based Video Inpainting Transformers Thiry, Guillaume Tang, Hao Timofte, Radu Van Gool, Luc Computer Vision and Pattern Recognition Video inpainting tasks have seen significant improvements in recent years with the rise of deep neural networks and, in particular, vision transformers. Although these models show promising reconstruction quality and temporal consistency, they are still unsuitable for live videos, one of the last steps to make them completely convincing and usable. The main limitations are that these state-of-the-art models inpaint using the whole video (offline processing) and show an insufficient frame rate. In our approach, we propose a framework to adapt existing inpainting transformers to these constraints by memorizing and refining redundant computations while maintaining a decent inpainting quality. Using this framework with some of the most recent inpainting models, we show great online results with a consistent throughput above 20 frames per second. The code and pretrained models will be made available upon acceptance. |
| title | Towards Online Real-Time Memory-based Video Inpainting Transformers |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.16161 |