ClustViT: Clustering-based Token Merging for Semantic Segmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909005620707328 |
|---|---|
| author | Montello, Fabio Güldenring, Ronja Nalpantidis, Lazaros |
| author_facet | Montello, Fabio Güldenring, Ronja Nalpantidis, Lazaros |
| contents | Vision Transformers can achieve high accuracy and strong generalization across various contexts, but their practical applicability on real-world robotic systems is limited due to their quadratic attention complexity. Recent works have focused on dynamically merging tokens according to the image complexity. Token merging works well for classification but is less suited to dense prediction. We propose ClustViT, where we expand upon the Vision Transformer (ViT) backbone and address semantic segmentation. Within our architecture, a trainable Cluster module merges similar tokens along the network guided by pseudo-clusters from segmentation masks. Subsequently, a Regenerator module restores fine details for downstream heads. Our approach achieves up to 2.18x fewer GFLOPs and 1.64x faster inference on three different datasets, with comparable segmentation accuracy. Our code and models will be made publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_01948 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ClustViT: Clustering-based Token Merging for Semantic Segmentation Montello, Fabio Güldenring, Ronja Nalpantidis, Lazaros Computer Vision and Pattern Recognition 68T45 I.2.10 Vision Transformers can achieve high accuracy and strong generalization across various contexts, but their practical applicability on real-world robotic systems is limited due to their quadratic attention complexity. Recent works have focused on dynamically merging tokens according to the image complexity. Token merging works well for classification but is less suited to dense prediction. We propose ClustViT, where we expand upon the Vision Transformer (ViT) backbone and address semantic segmentation. Within our architecture, a trainable Cluster module merges similar tokens along the network guided by pseudo-clusters from segmentation masks. Subsequently, a Regenerator module restores fine details for downstream heads. Our approach achieves up to 2.18x fewer GFLOPs and 1.64x faster inference on three different datasets, with comparable segmentation accuracy. Our code and models will be made publicly available. |
| title | ClustViT: Clustering-based Token Merging for Semantic Segmentation |
| topic | Computer Vision and Pattern Recognition 68T45 I.2.10 |
| url | https://arxiv.org/abs/2510.01948 |