OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913548404260864 |
|---|---|
| author | Chen, Lichang Hu, Hexiang Zhang, Mingda Chen, Yiwen Wang, Zifeng Li, Yandong Shyam, Pranav Zhou, Tianyi Huang, Heng Yang, Ming-Hsuan Gong, Boqing |
| author_facet | Chen, Lichang Hu, Hexiang Zhang, Mingda Chen, Yiwen Wang, Zifeng Li, Yandong Shyam, Pranav Zhou, Tianyi Huang, Heng Yang, Ming-Hsuan Gong, Boqing |
| contents | We introduce OmnixR, an evaluation suite designed to benchmark SoTA Omni-modality Language Models, such as GPT-4o and Gemini. Evaluating OLMs, which integrate multiple modalities such as text, vision, and audio, presents unique challenges. Particularly, the user message might often consist of multiple modalities, such that OLMs have to establish holistic understanding and reasoning across modalities to accomplish the task. Existing benchmarks are limited to single modality or dual-modality tasks, overlooking comprehensive multi-modal assessments of model reasoning. To address this, OmnixR offers two evaluation variants: (1)synthetic subset: a synthetic dataset generated automatically by translating text into multiple modalities--audio, images, video, and hybrids (Omnify). (2)realistic subset: a real-world dataset, manually curated and annotated by experts, for evaluating cross-modal reasoning in natural settings. OmnixR presents a unique evaluation towards assessing OLMs over a diverse mix of modalities, such as a question that involves video, audio, and text, providing a rigorous cross-modal reasoning testbed unlike any existing benchmarks. Our experiments find that all state-of-the-art OLMs struggle with OmnixR questions that require integrating information from multiple modalities to answer. Further analysis highlights differences in reasoning behavior, underscoring the challenges of omni-modal AI alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12219 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities Chen, Lichang Hu, Hexiang Zhang, Mingda Chen, Yiwen Wang, Zifeng Li, Yandong Shyam, Pranav Zhou, Tianyi Huang, Heng Yang, Ming-Hsuan Gong, Boqing Artificial Intelligence Computation and Language Multimedia We introduce OmnixR, an evaluation suite designed to benchmark SoTA Omni-modality Language Models, such as GPT-4o and Gemini. Evaluating OLMs, which integrate multiple modalities such as text, vision, and audio, presents unique challenges. Particularly, the user message might often consist of multiple modalities, such that OLMs have to establish holistic understanding and reasoning across modalities to accomplish the task. Existing benchmarks are limited to single modality or dual-modality tasks, overlooking comprehensive multi-modal assessments of model reasoning. To address this, OmnixR offers two evaluation variants: (1)synthetic subset: a synthetic dataset generated automatically by translating text into multiple modalities--audio, images, video, and hybrids (Omnify). (2)realistic subset: a real-world dataset, manually curated and annotated by experts, for evaluating cross-modal reasoning in natural settings. OmnixR presents a unique evaluation towards assessing OLMs over a diverse mix of modalities, such as a question that involves video, audio, and text, providing a rigorous cross-modal reasoning testbed unlike any existing benchmarks. Our experiments find that all state-of-the-art OLMs struggle with OmnixR questions that require integrating information from multiple modalities to answer. Further analysis highlights differences in reasoning behavior, underscoring the challenges of omni-modal AI alignment. |
| title | OmnixR: Evaluating Omni-modality Language Models on Reasoning across Modalities |
| topic | Artificial Intelligence Computation and Language Multimedia |
| url | https://arxiv.org/abs/2410.12219 |