LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915774750261248 |
|---|---|
| author | Fuller, Anthony Yassin, Yousef Wen, Junfeng Kyrollos, Daniel G. Ibrahim, Tarek Green, James R. Shelhamer, Evan |
| author_facet | Fuller, Anthony Yassin, Yousef Wen, Junfeng Kyrollos, Daniel G. Ibrahim, Tarek Green, James R. Shelhamer, Evan |
| contents | Vision transformers are ever larger, more accurate, and more expensive to compute. The expense is even more extreme at high resolution as the number of tokens grows quadratically with the image size. We turn to adaptive computation to cope with this cost by learning to predict where to compute. Our LookWhere method divides the computation between a low-resolution selector and a high-resolution extractor without ever processing the full high-resolution input. We jointly pretrain the selector and extractor without task supervision by distillation from a self-supervised teacher, in effect, learning where and what to compute simultaneously. Unlike prior token reduction methods, which pay to save by pruning already-computed tokens, and prior token selection methods, which require complex and expensive per-task optimization, LookWhere economically and accurately selects and extracts transferrable representations of images. We show that LookWhere excels at sparse recognition on high-resolution inputs (Traffic Signs), maintaining accuracy while reducing FLOPs by up to 34x and time by 6x. It also excels at standard recognition tasks that are global (ImageNet classification) or local (ADE20K segmentation), improving accuracy while reducing time by 1.36x. See https://github.com/antofuller/lookwhere for the code and weights. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18051 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision Fuller, Anthony Yassin, Yousef Wen, Junfeng Kyrollos, Daniel G. Ibrahim, Tarek Green, James R. Shelhamer, Evan Computer Vision and Pattern Recognition Vision transformers are ever larger, more accurate, and more expensive to compute. The expense is even more extreme at high resolution as the number of tokens grows quadratically with the image size. We turn to adaptive computation to cope with this cost by learning to predict where to compute. Our LookWhere method divides the computation between a low-resolution selector and a high-resolution extractor without ever processing the full high-resolution input. We jointly pretrain the selector and extractor without task supervision by distillation from a self-supervised teacher, in effect, learning where and what to compute simultaneously. Unlike prior token reduction methods, which pay to save by pruning already-computed tokens, and prior token selection methods, which require complex and expensive per-task optimization, LookWhere economically and accurately selects and extracts transferrable representations of images. We show that LookWhere excels at sparse recognition on high-resolution inputs (Traffic Signs), maintaining accuracy while reducing FLOPs by up to 34x and time by 6x. It also excels at standard recognition tasks that are global (ImageNet classification) or local (ADE20K segmentation), improving accuracy while reducing time by 1.36x. See https://github.com/antofuller/lookwhere for the code and weights. |
| title | LookWhere? Efficient Visual Recognition by Learning Where to Look and What to See from Self-Supervision |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.18051 |