Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.16599 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929644225167360 |
|---|---|
| author | Yin, Hang Lin, Zhifeng Liu, Xin Sun, Bin Li, Kan |
| author_facet | Yin, Hang Lin, Zhifeng Liu, Xin Sun, Bin Li, Kan |
| contents | Direction reasoning is essential for intelligent systems to understand the real world. While existing work focuses primarily on spatial reasoning, compass direction reasoning remains underexplored. To address this, we propose the Compass Direction Reasoning (CDR) benchmark, designed to evaluate the direction reasoning capabilities of multimodal language models (MLMs). CDR includes three types images to test spatial (up, down, left, right) and compass (north, south, east, west) directions. Our evaluation reveals that most MLMs struggle with direction reasoning, often performing at random guessing levels. Experiments show that training directly with CDR data yields limited improvements, as it requires an understanding of real-world physical rules. We explore the impact of mixdata and CoT fine-tuning methods, which significantly enhance MLM performance in compass direction reasoning by incorporating diverse data and step-by-step reasoning, improving the model's ability to understand direction relationships. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_16599 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Do Multimodal Language Models Really Understand Direction? A Benchmark for Compass Direction Reasoning Yin, Hang Lin, Zhifeng Liu, Xin Sun, Bin Li, Kan Artificial Intelligence Direction reasoning is essential for intelligent systems to understand the real world. While existing work focuses primarily on spatial reasoning, compass direction reasoning remains underexplored. To address this, we propose the Compass Direction Reasoning (CDR) benchmark, designed to evaluate the direction reasoning capabilities of multimodal language models (MLMs). CDR includes three types images to test spatial (up, down, left, right) and compass (north, south, east, west) directions. Our evaluation reveals that most MLMs struggle with direction reasoning, often performing at random guessing levels. Experiments show that training directly with CDR data yields limited improvements, as it requires an understanding of real-world physical rules. We explore the impact of mixdata and CoT fine-tuning methods, which significantly enhance MLM performance in compass direction reasoning by incorporating diverse data and step-by-step reasoning, improving the model's ability to understand direction relationships. |
| title | Do Multimodal Language Models Really Understand Direction? A Benchmark for Compass Direction Reasoning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2412.16599 |