CAT: Content-Adaptive Image Tokenization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916552649998336 |
|---|---|
| author | Shen, Junhong Tirumala, Kushal Yasunaga, Michihiro Misra, Ishan Zettlemoyer, Luke Yu, Lili Zhou, Chunting |
| author_facet | Shen, Junhong Tirumala, Kushal Yasunaga, Michihiro Misra, Ishan Zettlemoyer, Luke Yu, Lili Zhou, Chunting |
| contents | Most existing image tokenizers encode images into a fixed number of tokens or patches, overlooking the inherent variability in image complexity. To address this, we introduce Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design a caption-based evaluation system that leverages large language models (LLMs) to predict content complexity and determine the optimal compression ratio for a given image, taking into account factors critical to human perception. Trained on images with diverse compression ratios, CAT demonstrates robust performance in image reconstruction. We also utilize its variable-length latent representations to train Diffusion Transformers (DiTs) for ImageNet generation. By optimizing token allocation, CAT improves the FID score over fixed-ratio baselines trained with the same flops and boosts the inference throughput by 18.5%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_03120 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CAT: Content-Adaptive Image Tokenization Shen, Junhong Tirumala, Kushal Yasunaga, Michihiro Misra, Ishan Zettlemoyer, Luke Yu, Lili Zhou, Chunting Computer Vision and Pattern Recognition Most existing image tokenizers encode images into a fixed number of tokens or patches, overlooking the inherent variability in image complexity. To address this, we introduce Content-Adaptive Tokenizer (CAT), which dynamically adjusts representation capacity based on the image content and encodes simpler images into fewer tokens. We design a caption-based evaluation system that leverages large language models (LLMs) to predict content complexity and determine the optimal compression ratio for a given image, taking into account factors critical to human perception. Trained on images with diverse compression ratios, CAT demonstrates robust performance in image reconstruction. We also utilize its variable-length latent representations to train Diffusion Transformers (DiTs) for ImageNet generation. By optimizing token allocation, CAT improves the FID score over fixed-ratio baselines trained with the same flops and boosts the inference throughput by 18.5%. |
| title | CAT: Content-Adaptive Image Tokenization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.03120 |