AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915233718599680 |
|---|---|
| author | Chai, Wenhao Song, Enxin Du, Yilun Meng, Chenlin Madhavan, Vashisht Bar-Tal, Omer Hwang, Jenq-Neng Xie, Saining Manning, Christopher D. |
| author_facet | Chai, Wenhao Song, Enxin Du, Yilun Meng, Chenlin Madhavan, Vashisht Bar-Tal, Omer Hwang, Jenq-Neng Xie, Saining Manning, Christopher D. |
| contents | Video detailed captioning is a key task which aims to generate comprehensive and coherent textual descriptions of video content, benefiting both video understanding and generation. In this paper, we propose AuroraCap, a video captioner based on a large multimodal model. We follow the simplest architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, we implement the token merging strategy, reducing the number of input visual tokens. Surprisingly, we found that this strategy results in little performance loss. AuroraCap shows superior performance on various video and image captioning benchmarks, for example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include simple descriptions, consisting of a few dozen words, which limits research in this field. Therefore, we develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. In addition, we propose a new LLM-assisted metric VDCscore for bettering evaluation, which adopts a divide-and-conquer strategy to transform long caption evaluation into multiple short question-answer pairs. With the help of human Elo ranking, our experiments show that this benchmark better correlates with human judgments of video detailed captioning quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03051 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark Chai, Wenhao Song, Enxin Du, Yilun Meng, Chenlin Madhavan, Vashisht Bar-Tal, Omer Hwang, Jenq-Neng Xie, Saining Manning, Christopher D. Computer Vision and Pattern Recognition Video detailed captioning is a key task which aims to generate comprehensive and coherent textual descriptions of video content, benefiting both video understanding and generation. In this paper, we propose AuroraCap, a video captioner based on a large multimodal model. We follow the simplest architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, we implement the token merging strategy, reducing the number of input visual tokens. Surprisingly, we found that this strategy results in little performance loss. AuroraCap shows superior performance on various video and image captioning benchmarks, for example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include simple descriptions, consisting of a few dozen words, which limits research in this field. Therefore, we develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. In addition, we propose a new LLM-assisted metric VDCscore for bettering evaluation, which adopts a divide-and-conquer strategy to transform long caption evaluation into multiple short question-answer pairs. With the help of human Elo ranking, our experiments show that this benchmark better correlates with human judgments of video detailed captioning quality. |
| title | AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.03051 |