FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912972418318336 |
|---|---|
| author | Zhao, Peisen Zhang, Xiaopeng Xu, Mingxing Sun, Ruoyu Du, Zewei Wang, Dunzheng Zheng, Guanghao Xu, Haohang Zhang, Zhibo Zhang, Yuhang Ai, Yi Liu, Lin Tian, Qi |
| author_facet | Zhao, Peisen Zhang, Xiaopeng Xu, Mingxing Sun, Ruoyu Du, Zewei Wang, Dunzheng Zheng, Guanghao Xu, Haohang Zhang, Zhibo Zhang, Yuhang Ai, Yi Liu, Lin Tian, Qi |
| contents | While Multimodal Large Language Models (MLLMs) have experienced rapid advancements, their visual encoders frequently remain a performance bottleneck. Conventional CLIP-based encoders struggle with dense spatial tasks due to the loss of visual details caused by low-resolution pretraining and the reliance on noisy, coarse web-crawled image-text pairs. To overcome these limitations, we introduce FineViT, a novel vision encoder specifically designed to unlock fine-grained perception. By replacing coarse web data with dense recaptions, we systematically mitigate information loss through a progressive training paradigm.: first, the encoder is trained from scratch at a high native resolution on billions of global recaptioned image-text pairs, establishing a robust, detail rich semantic foundation. Subsequently, we further enhance its local perception through LLM alignment, utilizing our curated FineCap-450M dataset that comprises over $450$ million high quality local captions. Extensive experiments validate the effectiveness of the progressive strategy. FineViT achieves state-of-the-art zero-shot recognition and retrieval performance, especially in long-context retrieval, and consistently outperforms multimodal visual encoders such as SigLIP2 and Qwen-ViT when integrated into MLLMs. We hope FineViT could serve as a powerful new baseline for fine-grained visual perception. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_17326 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions Zhao, Peisen Zhang, Xiaopeng Xu, Mingxing Sun, Ruoyu Du, Zewei Wang, Dunzheng Zheng, Guanghao Xu, Haohang Zhang, Zhibo Zhang, Yuhang Ai, Yi Liu, Lin Tian, Qi Computer Vision and Pattern Recognition While Multimodal Large Language Models (MLLMs) have experienced rapid advancements, their visual encoders frequently remain a performance bottleneck. Conventional CLIP-based encoders struggle with dense spatial tasks due to the loss of visual details caused by low-resolution pretraining and the reliance on noisy, coarse web-crawled image-text pairs. To overcome these limitations, we introduce FineViT, a novel vision encoder specifically designed to unlock fine-grained perception. By replacing coarse web data with dense recaptions, we systematically mitigate information loss through a progressive training paradigm.: first, the encoder is trained from scratch at a high native resolution on billions of global recaptioned image-text pairs, establishing a robust, detail rich semantic foundation. Subsequently, we further enhance its local perception through LLM alignment, utilizing our curated FineCap-450M dataset that comprises over $450$ million high quality local captions. Extensive experiments validate the effectiveness of the progressive strategy. FineViT achieves state-of-the-art zero-shot recognition and retrieval performance, especially in long-context retrieval, and consistently outperforms multimodal visual encoders such as SigLIP2 and Qwen-ViT when integrated into MLLMs. We hope FineViT could serve as a powerful new baseline for fine-grained visual perception. |
| title | FineViT: Progressively Unlocking Fine-Grained Perception with Dense Recaptions |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.17326 |