Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.27322 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917450155556864 |
|---|---|
| author | Wu, Chenyang Lei, Lina Li, Fan Guo, Chun-Le Kong, Dehong Qin, Xinran Wang, Zhixin Cheng, Ming-Ming Li, Chongyi |
| author_facet | Wu, Chenyang Lei, Lina Li, Fan Guo, Chun-Le Kong, Dehong Qin, Xinran Wang, Zhixin Cheng, Ming-Ming Li, Chongyi |
| contents | Recent advances in Diffusion Transformer (DiT)-based video generation technologies have shown impressive results for video object removal. However, these methods still suffer from substantial inference latency. For instance, although MiniMax Remover achieves state-of-the-art visual quality, it operates at only around 10FPS, primarily due to dense computations over the entire spatiotemporal token space, even when only a small masked region actually requires processing. In this paper, we present YOSE, You Only Select Essential Tokens, an efficient fine-tuning framework. YOSE introduces two key components: Batch Variable-length Indexing (BVI) and Diffusion Process Simulator (DiffSim) Module. BVI is a differentiable dynamic indexing operator that adaptively selects essential tokens based on mask information, enabling variable-length token processing across samples. DiffSim provides a diffusion process approximation mechanism for unmasked tokens, which simulates the influence of unmasked regions within DiT self-attention to maintain semantic consistency for masked tokens. With these designs, YOSE achieves mask-aware acceleration, where the inference time scales approximately linearly with the masked regions, in contrast to full-token diffusion methods whose computation remains constant regardless of the mask size. Extensive experiments demonstrate that YOSE achieves up to 2.5X speedup in 70% of cases while maintaining visual quality comparable to the baseline. Code is available at: https://github.com/Wucy0519/YOSE-CVPR26. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_27322 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | YOSE: You Only Select Essential Tokens for Efficient DiT-based Video Object Removal Wu, Chenyang Lei, Lina Li, Fan Guo, Chun-Le Kong, Dehong Qin, Xinran Wang, Zhixin Cheng, Ming-Ming Li, Chongyi Computer Vision and Pattern Recognition Recent advances in Diffusion Transformer (DiT)-based video generation technologies have shown impressive results for video object removal. However, these methods still suffer from substantial inference latency. For instance, although MiniMax Remover achieves state-of-the-art visual quality, it operates at only around 10FPS, primarily due to dense computations over the entire spatiotemporal token space, even when only a small masked region actually requires processing. In this paper, we present YOSE, You Only Select Essential Tokens, an efficient fine-tuning framework. YOSE introduces two key components: Batch Variable-length Indexing (BVI) and Diffusion Process Simulator (DiffSim) Module. BVI is a differentiable dynamic indexing operator that adaptively selects essential tokens based on mask information, enabling variable-length token processing across samples. DiffSim provides a diffusion process approximation mechanism for unmasked tokens, which simulates the influence of unmasked regions within DiT self-attention to maintain semantic consistency for masked tokens. With these designs, YOSE achieves mask-aware acceleration, where the inference time scales approximately linearly with the masked regions, in contrast to full-token diffusion methods whose computation remains constant regardless of the mask size. Extensive experiments demonstrate that YOSE achieves up to 2.5X speedup in 70% of cases while maintaining visual quality comparable to the baseline. Code is available at: https://github.com/Wucy0519/YOSE-CVPR26. |
| title | YOSE: You Only Select Essential Tokens for Efficient DiT-based Video Object Removal |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2604.27322 |