Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.05737 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911819285659648 |
|---|---|
| author | Yu, Lijun Lezama, José Gundavarapu, Nitesh B. Versari, Luca Sohn, Kihyuk Minnen, David Cheng, Yong Birodkar, Vighnesh Gupta, Agrim Gu, Xiuye Hauptmann, Alexander G. Gong, Boqing Yang, Ming-Hsuan Essa, Irfan Ross, David A. Jiang, Lu |
| author_facet | Yu, Lijun Lezama, José Gundavarapu, Nitesh B. Versari, Luca Sohn, Kihyuk Minnen, David Cheng, Yong Birodkar, Vighnesh Gupta, Agrim Gu, Xiuye Hauptmann, Alexander G. Gong, Boqing Yang, Ming-Hsuan Essa, Irfan Ross, David A. Jiang, Lu |
| contents | While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation. To effectively use LLMs for visual generation, one crucial component is the visual tokenizer that maps pixel-space inputs to discrete tokens appropriate for LLM learning. In this paper, we introduce MAGVIT-v2, a video tokenizer designed to generate concise and expressive tokens for both videos and images using a common token vocabulary. Equipped with this new tokenizer, we show that LLMs outperform diffusion models on standard image and video generation benchmarks including ImageNet and Kinetics. In addition, we demonstrate that our tokenizer surpasses the previously top-performing video tokenizer on two more tasks: (1) video compression comparable to the next-generation video codec (VCC) according to human evaluations, and (2) learning effective representations for action recognition tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_05737 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation Yu, Lijun Lezama, José Gundavarapu, Nitesh B. Versari, Luca Sohn, Kihyuk Minnen, David Cheng, Yong Birodkar, Vighnesh Gupta, Agrim Gu, Xiuye Hauptmann, Alexander G. Gong, Boqing Yang, Ming-Hsuan Essa, Irfan Ross, David A. Jiang, Lu Computer Vision and Pattern Recognition Artificial Intelligence Multimedia While Large Language Models (LLMs) are the dominant models for generative tasks in language, they do not perform as well as diffusion models on image and video generation. To effectively use LLMs for visual generation, one crucial component is the visual tokenizer that maps pixel-space inputs to discrete tokens appropriate for LLM learning. In this paper, we introduce MAGVIT-v2, a video tokenizer designed to generate concise and expressive tokens for both videos and images using a common token vocabulary. Equipped with this new tokenizer, we show that LLMs outperform diffusion models on standard image and video generation benchmarks including ImageNet and Kinetics. In addition, we demonstrate that our tokenizer surpasses the previously top-performing video tokenizer on two more tasks: (1) video compression comparable to the next-generation video codec (VCC) according to human evaluations, and (2) learning effective representations for action recognition tasks. |
| title | Language Model Beats Diffusion -- Tokenizer is Key to Visual Generation |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Multimedia |
| url | https://arxiv.org/abs/2310.05737 |