PolySmart @ TRECVid 2024 Medical Video Question Answering
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913620800045056 |
|---|---|
| author | Wu, Jiaxin Jiang, Yiyang Wei, Xiao-Yong Li, Qing |
| author_facet | Wu, Jiaxin Jiang, Yiyang Wei, Xiao-Yong Li, Qing |
| contents | Video Corpus Visual Answer Localization (VCVAL) includes question-related video retrieval and visual answer localization in the videos. Specifically, we use text-to-text retrieval to find relevant videos for a medical question based on the similarity of video transcript and answers generated by GPT4. For the visual answer localization, the start and end timestamps of the answer are predicted by the alignments on both visual content and subtitles with queries. For the Query-Focused Instructional Step Captioning (QFISC) task, the step captions are generated by GPT4. Specifically, we provide the video captions generated by the LLaVA-Next-Video model and the video subtitles with timestamps as context, and ask GPT4 to generate step captions for the given medical query. We only submit one run for evaluation and it obtains a F-score of 11.92 and mean IoU of 9.6527. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_15514 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PolySmart @ TRECVid 2024 Medical Video Question Answering Wu, Jiaxin Jiang, Yiyang Wei, Xiao-Yong Li, Qing Computer Vision and Pattern Recognition Multimedia Video Corpus Visual Answer Localization (VCVAL) includes question-related video retrieval and visual answer localization in the videos. Specifically, we use text-to-text retrieval to find relevant videos for a medical question based on the similarity of video transcript and answers generated by GPT4. For the visual answer localization, the start and end timestamps of the answer are predicted by the alignments on both visual content and subtitles with queries. For the Query-Focused Instructional Step Captioning (QFISC) task, the step captions are generated by GPT4. Specifically, we provide the video captions generated by the LLaVA-Next-Video model and the video subtitles with timestamps as context, and ask GPT4 to generate step captions for the given medical query. We only submit one run for evaluation and it obtains a F-score of 11.92 and mean IoU of 9.6527. |
| title | PolySmart @ TRECVid 2024 Medical Video Question Answering |
| topic | Computer Vision and Pattern Recognition Multimedia |
| url | https://arxiv.org/abs/2412.15514 |