Describe Anything: Detailed Localized Image and Video Captioning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912341201780736 |
|---|---|
| author | Lian, Long Ding, Yifan Ge, Yunhao Liu, Sifei Mao, Hanzi Li, Boyi Pavone, Marco Liu, Ming-Yu Darrell, Trevor Yala, Adam Cui, Yin |
| author_facet | Lian, Long Ding, Yifan Ge, Yunhao Liu, Sifei Mao, Hanzi Li, Boyi Pavone, Marco Liu, Ming-Yu Darrell, Trevor Yala, Adam Cui, Yin |
| contents | Generating detailed and accurate descriptions for specific regions in images and videos remains a fundamental challenge for vision-language models. We introduce the Describe Anything Model (DAM), a model designed for detailed localized captioning (DLC). DAM preserves both local details and global context through two key innovations: a focal prompt, which ensures high-resolution encoding of targeted regions, and a localized vision backbone, which integrates precise localization with its broader context. To tackle the scarcity of high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark designed to evaluate DLC without relying on reference captions. DAM sets new state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_16072 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Describe Anything: Detailed Localized Image and Video Captioning Lian, Long Ding, Yifan Ge, Yunhao Liu, Sifei Mao, Hanzi Li, Boyi Pavone, Marco Liu, Ming-Yu Darrell, Trevor Yala, Adam Cui, Yin Computer Vision and Pattern Recognition Artificial Intelligence Generating detailed and accurate descriptions for specific regions in images and videos remains a fundamental challenge for vision-language models. We introduce the Describe Anything Model (DAM), a model designed for detailed localized captioning (DLC). DAM preserves both local details and global context through two key innovations: a focal prompt, which ensures high-resolution encoding of targeted regions, and a localized vision backbone, which integrates precise localization with its broader context. To tackle the scarcity of high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark designed to evaluate DLC without relying on reference captions. DAM sets new state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and detailed multi-sentence localized image and video captioning. |
| title | Describe Anything: Detailed Localized Image and Video Captioning |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2504.16072 |