Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917637545525248 |
|---|---|
| author | Zhang, Haotian You, Haoxuan Dufter, Philipp Zhang, Bowen Chen, Chen Chen, Hong-You Fu, Tsu-Jui Wang, William Yang Chang, Shih-Fu Gan, Zhe Yang, Yinfei |
| author_facet | Zhang, Haotian You, Haoxuan Dufter, Philipp Zhang, Bowen Chen, Chen Chen, Hong-You Fu, Tsu-Jui Wang, William Yang Chang, Shih-Fu Gan, Zhe Yang, Yinfei |
| contents | While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual encoder and failed to perform well on broader tasks. In this work, we unveil Ferret-v2, a significant upgrade to Ferret, with three key designs. (1) Any resolution grounding and referring: A flexible approach that effortlessly handles higher image resolution, improving the model's ability to process and understand images in greater detail. (2) Multi-granularity visual encoding: By integrating the additional DINOv2 encoder, the model learns better and diverse underlying contexts for global and fine-grained visual information. (3) A three-stage training paradigm: Besides image-caption alignment, an additional stage is proposed for high-resolution dense alignment before the final instruction tuning. Experiments show that Ferret-v2 provides substantial improvements over Ferret and other state-of-the-art methods, thanks to its high-resolution scaling and fine-grained visual processing. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_07973 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models Zhang, Haotian You, Haoxuan Dufter, Philipp Zhang, Bowen Chen, Chen Chen, Hong-You Fu, Tsu-Jui Wang, William Yang Chang, Shih-Fu Gan, Zhe Yang, Yinfei Computer Vision and Pattern Recognition While Ferret seamlessly integrates regional understanding into the Large Language Model (LLM) to facilitate its referring and grounding capability, it poses certain limitations: constrained by the pre-trained fixed visual encoder and failed to perform well on broader tasks. In this work, we unveil Ferret-v2, a significant upgrade to Ferret, with three key designs. (1) Any resolution grounding and referring: A flexible approach that effortlessly handles higher image resolution, improving the model's ability to process and understand images in greater detail. (2) Multi-granularity visual encoding: By integrating the additional DINOv2 encoder, the model learns better and diverse underlying contexts for global and fine-grained visual information. (3) A three-stage training paradigm: Besides image-caption alignment, an additional stage is proposed for high-resolution dense alignment before the final instruction tuning. Experiments show that Ferret-v2 provides substantial improvements over Ferret and other state-of-the-art methods, thanks to its high-resolution scaling and fine-grained visual processing. |
| title | Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.07973 |