CoT Referring: Improving Referring Expression Tasks with Grounded Reasoning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911381312241664 |
|---|---|
| author | Dong, Qihua Figueroa, Luis Zhao, Handong Kafle, Kushal Kuen, Jason Ding, Zhihong Cohen, Scott Fu, Yun |
| author_facet | Dong, Qihua Figueroa, Luis Zhao, Handong Kafle, Kushal Kuen, Jason Ding, Zhihong Cohen, Scott Fu, Yun |
| contents | Referring Expression Comprehension and Segmentation are critical tasks for assessing the integration of language understanding and image comprehension, serving as benchmarks for Multimodal Large Language Models (MLLMs) capabilities. To address these challenges, we propose a new strategy, CoT Referring, which enhances model reasoning across modalities through a structured, chain-of-thought training data structure. Our approach systematically parses textual structures to a sequential referring step, where in each step it identifies relationships and ensures consistent reference alignment, thereby improving accuracy in complex query scenarios. We restructure the training data to enforce a new output form, providing new annotations for existing datasets and compiling an evaluation benchmark from existing resources. This benchmark is designed explicitly for complex referring cases. We also integrate detection and segmentation capabilities into a unified MLLM framework, training it with a novel adaptive weighted loss to optimize performance. Experimental results on our curated benchmark and RefCOCO/+/g demonstrate the effectiveness of our approach, with a notable increase of 2.5%+ over baseline models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_06243 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CoT Referring: Improving Referring Expression Tasks with Grounded Reasoning Dong, Qihua Figueroa, Luis Zhao, Handong Kafle, Kushal Kuen, Jason Ding, Zhihong Cohen, Scott Fu, Yun Computation and Language Artificial Intelligence Referring Expression Comprehension and Segmentation are critical tasks for assessing the integration of language understanding and image comprehension, serving as benchmarks for Multimodal Large Language Models (MLLMs) capabilities. To address these challenges, we propose a new strategy, CoT Referring, which enhances model reasoning across modalities through a structured, chain-of-thought training data structure. Our approach systematically parses textual structures to a sequential referring step, where in each step it identifies relationships and ensures consistent reference alignment, thereby improving accuracy in complex query scenarios. We restructure the training data to enforce a new output form, providing new annotations for existing datasets and compiling an evaluation benchmark from existing resources. This benchmark is designed explicitly for complex referring cases. We also integrate detection and segmentation capabilities into a unified MLLM framework, training it with a novel adaptive weighted loss to optimize performance. Experimental results on our curated benchmark and RefCOCO/+/g demonstrate the effectiveness of our approach, with a notable increase of 2.5%+ over baseline models. |
| title | CoT Referring: Improving Referring Expression Tasks with Grounded Reasoning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.06243 |