Heptapod: Language Modeling on Visual Signals
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916996047699968 |
|---|---|
| author | Zhu, Yongxin Chen, Jiawei Chen, Yuanzhe Chen, Zhuo Jia, Dongya Cong, Jian Zhuang, Xiaobin Wang, Yuping Wang, Yuxuan |
| author_facet | Zhu, Yongxin Chen, Jiawei Chen, Yuanzhe Chen, Zhuo Jia, Dongya Cong, Jian Zhuang, Xiaobin Wang, Yuping Wang, Yuxuan |
| contents | We introduce Heptapod, an image autoregressive model that adheres to the foundational principles of language modeling. Heptapod employs \textbf{causal attention}, \textbf{eliminates reliance on CFG}, and \textbf{eschews the trend of semantic tokenizers}. Our key innovation is \textit{next 2D distribution prediction}: a causal Transformer with reconstruction-focused visual tokenizer, learns to predict the distribution over the entire 2D spatial grid of images at each timestep. This learning objective unifies the sequential modeling of autoregressive framework with the holistic self-supervised learning of masked autoencoding, enabling the model to capture comprehensive image semantics via generative training. On the ImageNet generation benchmark, Heptapod achieves an FID of $2.70$, significantly outperforming previous causal autoregressive approaches. We hope our work inspires a principled rethinking of language modeling on visual signals and beyond. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_06673 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Heptapod: Language Modeling on Visual Signals Zhu, Yongxin Chen, Jiawei Chen, Yuanzhe Chen, Zhuo Jia, Dongya Cong, Jian Zhuang, Xiaobin Wang, Yuping Wang, Yuxuan Computer Vision and Pattern Recognition Artificial Intelligence We introduce Heptapod, an image autoregressive model that adheres to the foundational principles of language modeling. Heptapod employs \textbf{causal attention}, \textbf{eliminates reliance on CFG}, and \textbf{eschews the trend of semantic tokenizers}. Our key innovation is \textit{next 2D distribution prediction}: a causal Transformer with reconstruction-focused visual tokenizer, learns to predict the distribution over the entire 2D spatial grid of images at each timestep. This learning objective unifies the sequential modeling of autoregressive framework with the holistic self-supervised learning of masked autoencoding, enabling the model to capture comprehensive image semantics via generative training. On the ImageNet generation benchmark, Heptapod achieves an FID of $2.70$, significantly outperforming previous causal autoregressive approaches. We hope our work inspires a principled rethinking of language modeling on visual signals and beyond. |
| title | Heptapod: Language Modeling on Visual Signals |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2510.06673 |