VidEoMT: Your ViT is Secretly Also a Video Segmentation Model
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910040831557632 |
|---|---|
| author | Norouzi, Narges Zulfikar, Idil Esen Cavagnero, Niccolò Kerssies, Tommie Leibe, Bastian Dubbelman, Gijs de Geus, Daan |
| author_facet | Norouzi, Narges Zulfikar, Idil Esen Cavagnero, Niccolò Kerssies, Tommie Leibe, Bastian Dubbelman, Gijs de Geus, Daan |
| contents | Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x-10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_17807 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | VidEoMT: Your ViT is Secretly Also a Video Segmentation Model Norouzi, Narges Zulfikar, Idil Esen Cavagnero, Niccolò Kerssies, Tommie Leibe, Bastian Dubbelman, Gijs de Geus, Daan Computer Vision and Pattern Recognition Existing online video segmentation models typically combine a per-frame segmenter with complex specialized tracking modules. While effective, these modules introduce significant architectural complexity and computational overhead. Recent studies suggest that plain Vision Transformer (ViT) encoders, when scaled with sufficient capacity and large-scale pre-training, can conduct accurate image segmentation without requiring specialized modules. Motivated by this observation, we propose the Video Encoder-only Mask Transformer (VidEoMT), a simple encoder-only video segmentation model that eliminates the need for dedicated tracking modules. To enable temporal modeling in an encoder-only ViT, VidEoMT introduces a lightweight query propagation mechanism that carries information across frames by reusing queries from the previous frame. To balance this with adaptability to new content, it employs a query fusion strategy that combines the propagated queries with a set of temporally-agnostic learned queries. As a result, VidEoMT attains the benefits of a tracker without added complexity, achieving competitive accuracy while being 5x-10x faster, running at up to 160 FPS with a ViT-L backbone. Code: https://www.tue-mps.org/videomt/ |
| title | VidEoMT: Your ViT is Secretly Also a Video Segmentation Model |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2602.17807 |