Object Recognition as Next Token Prediction
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917626668646400 |
|---|---|
| author | Yue, Kaiyu Chen, Bor-Chun Geiping, Jonas Li, Hengduo Goldstein, Tom Lim, Ser-Nam |
| author_facet | Yue, Kaiyu Chen, Bor-Chun Geiping, Jonas Li, Hengduo Goldstein, Tom Lim, Ser-Nam |
| contents | We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embeddings to form labels. To ground this prediction process in auto-regression, we customize a non-causal attention mask for the decoder, incorporating two key features: modeling tokens from different labels to be independent, and treating image tokens as a prefix. This masking mechanism inspires an efficient method - one-shot sampling - to simultaneously sample tokens of multiple labels in parallel and rank generated labels by their probabilities during inference. To further enhance the efficiency, we propose a simple strategy to construct a compact decoder by simply discarding the intermediate blocks of a pretrained language model. This approach yields a decoder that matches the full model's performance while being notably more efficient. The code is available at https://github.com/kaiyuyue/nxtp |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_02142 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Object Recognition as Next Token Prediction Yue, Kaiyu Chen, Bor-Chun Geiping, Jonas Li, Hengduo Goldstein, Tom Lim, Ser-Nam Computer Vision and Pattern Recognition We present an approach to pose object recognition as next token prediction. The idea is to apply a language decoder that auto-regressively predicts the text tokens from image embeddings to form labels. To ground this prediction process in auto-regression, we customize a non-causal attention mask for the decoder, incorporating two key features: modeling tokens from different labels to be independent, and treating image tokens as a prefix. This masking mechanism inspires an efficient method - one-shot sampling - to simultaneously sample tokens of multiple labels in parallel and rank generated labels by their probabilities during inference. To further enhance the efficiency, we propose a simple strategy to construct a compact decoder by simply discarding the intermediate blocks of a pretrained language model. This approach yields a decoder that matches the full model's performance while being notably more efficient. The code is available at https://github.com/kaiyuyue/nxtp |
| title | Object Recognition as Next Token Prediction |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.02142 |