StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912951705796608 |
|---|---|
| author | Nguyen, Duy M. H. Tran, Tuan A. Nguyen, Duong Xie, Siwei Nguyen, Trung Q. Truong, Mai T. N. Palenicek, Daniel Le, An T. Barz, Michael Nguyen, TrungTin Dam, Tuan Le, Ngan Vu, Minh Doan, Khoa Ngo, Vien Xie, Pengtao Zou, James Sonntag, Daniel Peters, Jan Niepert, Mathias |
| author_facet | Nguyen, Duy M. H. Tran, Tuan A. Nguyen, Duong Xie, Siwei Nguyen, Trung Q. Truong, Mai T. N. Palenicek, Daniel Le, An T. Barz, Michael Nguyen, TrungTin Dam, Tuan Le, Ngan Vu, Minh Doan, Khoa Ngo, Vien Xie, Pengtao Zou, James Sonntag, Daniel Peters, Jan Niepert, Mathias |
| contents | Recent token merging techniques for Vision Transformers (ViTs) provide substantial speedups by reducing the number of tokens processed by self-attention, often without retraining. However, their direct application to the Segment Anything Model (SAM) family is nontrivial: SAM's image encoder mixes windowed and global attention, and its mask decoder relies on dense, prompt-conditioned features for precise boundary prediction. We systematically evaluate representative token-merging methods on SAM and Medical SAM in a strict off-the-shelf setting, and find that existing destination-selection heuristics can erode boundaries and leak prompt information as merge rates increase. We propose \textbf{StructSAM}, a resolution-preserving merge-unmerge framework tailored to SAM. StructSAM computes a lightweight token-energy score from first-order feature gradients, uses grid-based flatness screening to protect boundary and prompt regions, and merges tokens within flat areas toward low-energy destinations with explicit token recovery. We further provide a spectral graph coarsening view showing that score-guided merging yields bounded Laplacian spectral distortion compared to random or window-restricted baselines. Across eight natural and medical benchmarks, StructSAM reduces encoder FLOPs by 25-30\% (up to 40\%+ with prompt-aware merging) with minor drops in mIoU/Dice, consistently outperforming ToMe, PiToMe, ToMeSD, VidToMe, and ALGM at the same compute. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_07307 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models Nguyen, Duy M. H. Tran, Tuan A. Nguyen, Duong Xie, Siwei Nguyen, Trung Q. Truong, Mai T. N. Palenicek, Daniel Le, An T. Barz, Michael Nguyen, TrungTin Dam, Tuan Le, Ngan Vu, Minh Doan, Khoa Ngo, Vien Xie, Pengtao Zou, James Sonntag, Daniel Peters, Jan Niepert, Mathias Computer Vision and Pattern Recognition Machine Learning Recent token merging techniques for Vision Transformers (ViTs) provide substantial speedups by reducing the number of tokens processed by self-attention, often without retraining. However, their direct application to the Segment Anything Model (SAM) family is nontrivial: SAM's image encoder mixes windowed and global attention, and its mask decoder relies on dense, prompt-conditioned features for precise boundary prediction. We systematically evaluate representative token-merging methods on SAM and Medical SAM in a strict off-the-shelf setting, and find that existing destination-selection heuristics can erode boundaries and leak prompt information as merge rates increase. We propose \textbf{StructSAM}, a resolution-preserving merge-unmerge framework tailored to SAM. StructSAM computes a lightweight token-energy score from first-order feature gradients, uses grid-based flatness screening to protect boundary and prompt regions, and merges tokens within flat areas toward low-energy destinations with explicit token recovery. We further provide a spectral graph coarsening view showing that score-guided merging yields bounded Laplacian spectral distortion compared to random or window-restricted baselines. Across eight natural and medical benchmarks, StructSAM reduces encoder FLOPs by 25-30\% (up to 40\%+ with prompt-aware merging) with minor drops in mIoU/Dice, consistently outperforming ToMe, PiToMe, ToMeSD, VidToMe, and ALGM at the same compute. |
| title | StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2603.07307 |