A Spitting Image: Modular Superpixel Tokenization in Vision Transformers
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914160627941376 |
|---|---|
| author | Aasan, Marius Kolbjørnsen, Odd Solberg, Anne Schistad Rivera, Adín Ramirez |
| author_facet | Aasan, Marius Kolbjørnsen, Odd Solberg, Anne Schistad Rivera, Adín Ramirez |
| contents | Vision Transformer (ViT) architectures traditionally employ a grid-based approach to tokenization independent of the semantic content of an image. We propose a modular superpixel tokenization strategy which decouples tokenization and feature extraction; a shift from contemporary approaches where these are treated as an undifferentiated whole. Using on-line content-aware tokenization and scale- and shape-invariant positional embeddings, we perform experiments and ablations that contrast our approach with patch-based tokenization and randomized partitions as baselines. We show that our method significantly improves the faithfulness of attributions, gives pixel-level granularity on zero-shot unsupervised dense prediction tasks, while maintaining predictive performance in classification tasks. Our approach provides a modular tokenization framework commensurable with standard architectures, extending the space of ViTs to a larger class of semantically-rich models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_07680 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Spitting Image: Modular Superpixel Tokenization in Vision Transformers Aasan, Marius Kolbjørnsen, Odd Solberg, Anne Schistad Rivera, Adín Ramirez Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning 68T45 I.2.10; I.4.10 Vision Transformer (ViT) architectures traditionally employ a grid-based approach to tokenization independent of the semantic content of an image. We propose a modular superpixel tokenization strategy which decouples tokenization and feature extraction; a shift from contemporary approaches where these are treated as an undifferentiated whole. Using on-line content-aware tokenization and scale- and shape-invariant positional embeddings, we perform experiments and ablations that contrast our approach with patch-based tokenization and randomized partitions as baselines. We show that our method significantly improves the faithfulness of attributions, gives pixel-level granularity on zero-shot unsupervised dense prediction tasks, while maintaining predictive performance in classification tasks. Our approach provides a modular tokenization framework commensurable with standard architectures, extending the space of ViTs to a larger class of semantically-rich models. |
| title | A Spitting Image: Modular Superpixel Tokenization in Vision Transformers |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning 68T45 I.2.10; I.4.10 |
| url | https://arxiv.org/abs/2408.07680 |