Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokens
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916661487992832 |
|---|---|
| author | Lew, Jaihyun Jang, Soohyuk Lee, Jaehoon Yoo, Seungryong Kim, Eunji Lee, Saehyung Mok, Jisoo Kim, Siwon Yoon, Sungroh |
| author_facet | Lew, Jaihyun Jang, Soohyuk Lee, Jaehoon Yoo, Seungryong Kim, Eunji Lee, Saehyung Mok, Jisoo Kim, Siwon Yoon, Sungroh |
| contents | Transformers, a groundbreaking architecture proposed for Natural Language Processing (NLP), have also achieved remarkable success in Computer Vision. A cornerstone of their success lies in the attention mechanism, which models relationships among tokens. While the tokenization process in NLP inherently ensures that a single token does not contain multiple semantics, the tokenization of Vision Transformer (ViT) utilizes tokens from uniformly partitioned square image patches, which may result in an arbitrary mixing of visual concepts in a token. In this work, we propose to substitute the grid-based tokenization in ViT with superpixel tokenization, which employs superpixels to generate a token that encapsulates a sole visual concept. Unfortunately, the diverse shapes, sizes, and locations of superpixels make integrating superpixels into ViT tokenization rather challenging. Our tokenization pipeline, comprised of pre-aggregate extraction and superpixel-aware aggregation, overcomes the challenges that arise in superpixel tokenization. Extensive experiments demonstrate that our approach, which exhibits strong compatibility with existing frameworks, enhances the accuracy and robustness of ViT on various downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_04680 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokens Lew, Jaihyun Jang, Soohyuk Lee, Jaehoon Yoo, Seungryong Kim, Eunji Lee, Saehyung Mok, Jisoo Kim, Siwon Yoon, Sungroh Computer Vision and Pattern Recognition Transformers, a groundbreaking architecture proposed for Natural Language Processing (NLP), have also achieved remarkable success in Computer Vision. A cornerstone of their success lies in the attention mechanism, which models relationships among tokens. While the tokenization process in NLP inherently ensures that a single token does not contain multiple semantics, the tokenization of Vision Transformer (ViT) utilizes tokens from uniformly partitioned square image patches, which may result in an arbitrary mixing of visual concepts in a token. In this work, we propose to substitute the grid-based tokenization in ViT with superpixel tokenization, which employs superpixels to generate a token that encapsulates a sole visual concept. Unfortunately, the diverse shapes, sizes, and locations of superpixels make integrating superpixels into ViT tokenization rather challenging. Our tokenization pipeline, comprised of pre-aggregate extraction and superpixel-aware aggregation, overcomes the challenges that arise in superpixel tokenization. Extensive experiments demonstrate that our approach, which exhibits strong compatibility with existing frameworks, enhances the accuracy and robustness of ViT on various downstream tasks. |
| title | Superpixel Tokenization for Vision Transformers: Preserving Semantic Integrity in Visual Tokens |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.04680 |