TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917799376453632 |
|---|---|
| author | Sanders, Kate Weir, Nathaniel Van Durme, Benjamin |
| author_facet | Sanders, Kate Weir, Nathaniel Van Durme, Benjamin |
| contents | It is challenging for models to understand complex, multimodal content such as television clips, and this is in part because video-language models often rely on single-modality reasoning and lack interpretability. To combat these issues we propose TV-TREES, the first multimodal entailment tree generator. TV-TREES serves as an approach to video understanding that promotes interpretable joint-modality reasoning by searching for trees of entailment relationships between simple text-video evidence and higher-level conclusions that prove question-answer pairs. We also introduce the task of multimodal entailment tree generation to evaluate reasoning quality. Our method's performance on the challenging TVQA benchmark demonstrates interpretable, state-of-the-art zero-shot performance on full clips, illustrating that multimodal entailment tree generation can be a best-of-both-worlds alternative to black-box systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_19467 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning Sanders, Kate Weir, Nathaniel Van Durme, Benjamin Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition I.2.7; I.2.10 It is challenging for models to understand complex, multimodal content such as television clips, and this is in part because video-language models often rely on single-modality reasoning and lack interpretability. To combat these issues we propose TV-TREES, the first multimodal entailment tree generator. TV-TREES serves as an approach to video understanding that promotes interpretable joint-modality reasoning by searching for trees of entailment relationships between simple text-video evidence and higher-level conclusions that prove question-answer pairs. We also introduce the task of multimodal entailment tree generation to evaluate reasoning quality. Our method's performance on the challenging TVQA benchmark demonstrates interpretable, state-of-the-art zero-shot performance on full clips, illustrating that multimodal entailment tree generation can be a best-of-both-worlds alternative to black-box systems. |
| title | TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition I.2.7; I.2.10 |
| url | https://arxiv.org/abs/2402.19467 |