Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2505.17011 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915552177422336 |
|---|---|
| author | Li, Yan Tian, Changyao Xia, Renqiu Liao, Ning Guo, Weiwei Yan, Junchi Li, Hongsheng Dai, Jifeng Li, Hao Yang, Xue |
| author_facet | Li, Yan Tian, Changyao Xia, Renqiu Liao, Ning Guo, Weiwei Yan, Junchi Li, Hongsheng Dai, Jifeng Li, Hao Yang, Xue |
| contents | We propose AdapTok, an adaptive temporal causal video tokenizer that can flexibly allocate tokens for different frames based on video content. AdapTok is equipped with a block-wise masking strategy that randomly drops tail tokens of each block during training, and a block causal scorer to predict the reconstruction quality of video frames using different numbers of tokens. During inference, an adaptive token allocation strategy based on integer linear programming is further proposed to adjust token usage given predicted scores. Such design allows for sample-wise, content-aware, and temporally dynamic token allocation under a controllable overall budget. Extensive experiments for video reconstruction and generation on UCF-101 and Kinetics-600 demonstrate the effectiveness of our approach. Without additional image data, AdapTok consistently improves reconstruction quality and generation performance under different token budgets, allowing for more scalable and token-efficient generative video modeling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17011 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space Li, Yan Tian, Changyao Xia, Renqiu Liao, Ning Guo, Weiwei Yan, Junchi Li, Hongsheng Dai, Jifeng Li, Hao Yang, Xue Computer Vision and Pattern Recognition We propose AdapTok, an adaptive temporal causal video tokenizer that can flexibly allocate tokens for different frames based on video content. AdapTok is equipped with a block-wise masking strategy that randomly drops tail tokens of each block during training, and a block causal scorer to predict the reconstruction quality of video frames using different numbers of tokens. During inference, an adaptive token allocation strategy based on integer linear programming is further proposed to adjust token usage given predicted scores. Such design allows for sample-wise, content-aware, and temporally dynamic token allocation under a controllable overall budget. Extensive experiments for video reconstruction and generation on UCF-101 and Kinetics-600 demonstrate the effectiveness of our approach. Without additional image data, AdapTok consistently improves reconstruction quality and generation performance under different token budgets, allowing for more scalable and token-efficient generative video modeling. |
| title | Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.17011 |