Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities?
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913880129667072 |
|---|---|
| author | Jin, Jiahe He, Yanheng Yang, Mingyan |
| author_facet | Jin, Jiahe He, Yanheng Yang, Mingyan |
| contents | In this work, we identify the "2D-Cheating" problem in 3D LLM evaluation, where these tasks might be easily solved by VLMs with rendered images of point clouds, exposing ineffective evaluation of 3D LLMs' unique 3D capabilities. We test VLM performance across multiple 3D LLM benchmarks and, using this as a reference, propose principles for better assessing genuine 3D understanding. We also advocate explicitly separating 3D abilities from 1D or 2D aspects when evaluating 3D LLMs. Code and data are available at https://github.com/LLM-class-group/Revisiting-3D-LLM-Benchmarks |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08503 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities? Jin, Jiahe He, Yanheng Yang, Mingyan Artificial Intelligence In this work, we identify the "2D-Cheating" problem in 3D LLM evaluation, where these tasks might be easily solved by VLMs with rendered images of point clouds, exposing ineffective evaluation of 3D LLMs' unique 3D capabilities. We test VLM performance across multiple 3D LLM benchmarks and, using this as a reference, propose principles for better assessing genuine 3D understanding. We also advocate explicitly separating 3D abilities from 1D or 2D aspects when evaluating 3D LLMs. Code and data are available at https://github.com/LLM-class-group/Revisiting-3D-LLM-Benchmarks |
| title | Revisiting 3D LLM Benchmarks: Are We Really Testing 3D Capabilities? |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2502.08503 |