What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916029774430208 |
|---|---|
| author | Yuan, Zhenlong Wang, Yue Zhang, Dapeng Cui, Kejin Chen, Rui Tang, Jing Sun, Lei Yu, Hongwei Qian, Chengxuan Chu, Xiangxiang Li, Shuo Zhou, Yuyin |
| author_facet | Yuan, Zhenlong Wang, Yue Zhang, Dapeng Cui, Kejin Chen, Rui Tang, Jing Sun, Lei Yu, Hongwei Qian, Chengxuan Chu, Xiangxiang Li, Shuo Zhou, Yuyin |
| contents | Multimodal Large Language Models have shown promising capabilities in bridging visual and textual reasoning, yet their reasoning capabilities in Open-Vocabulary Human-Object Interaction (OV-HOI) are limited by cross-modal hallucinations and limited viewpoints of images. To address this, we propose ImagineAgent, an agentic framework that integrates cognitive mapping, tool-augmented reinforcement learning (RL), and generative world modeling for robust OV-HOI understanding. Specifically, we first propose an innovative CoT dataset named hicodet-6K for supervised fine-tuning (SFT), which effectively bridges the perception-to-cognition gap by structuring perceived entities into interaction pairs for comprehensive predictions. Subsequently, we develop a multimodal tool library integrating online retrieval, image cropping, and generative modeling, enabling the agent to dynamically augment reasoning with domain-specific tools to resolve visual-semantic ambiguities and hallucinations during inference. Moreover, we incorporate a generative model to reconstruct alternative viewpoints, enabling the agent to 'imagine' under limited viewpoints. Finally, we propose a composite reward mechanism to jointly optimize prediction accuracy and tool efficiency. Evaluations on both SWIG-HOI and HICO-DET datasets demonstrate that our method achieves state-of-the-art performance while requiring merely 36.7% of the training data compared to existing methods, validating our robustness, empirical effectiveness and efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_11499 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation Yuan, Zhenlong Wang, Yue Zhang, Dapeng Cui, Kejin Chen, Rui Tang, Jing Sun, Lei Yu, Hongwei Qian, Chengxuan Chu, Xiangxiang Li, Shuo Zhou, Yuyin Computer Vision and Pattern Recognition Multimodal Large Language Models have shown promising capabilities in bridging visual and textual reasoning, yet their reasoning capabilities in Open-Vocabulary Human-Object Interaction (OV-HOI) are limited by cross-modal hallucinations and limited viewpoints of images. To address this, we propose ImagineAgent, an agentic framework that integrates cognitive mapping, tool-augmented reinforcement learning (RL), and generative world modeling for robust OV-HOI understanding. Specifically, we first propose an innovative CoT dataset named hicodet-6K for supervised fine-tuning (SFT), which effectively bridges the perception-to-cognition gap by structuring perceived entities into interaction pairs for comprehensive predictions. Subsequently, we develop a multimodal tool library integrating online retrieval, image cropping, and generative modeling, enabling the agent to dynamically augment reasoning with domain-specific tools to resolve visual-semantic ambiguities and hallucinations during inference. Moreover, we incorporate a generative model to reconstruct alternative viewpoints, enabling the agent to 'imagine' under limited viewpoints. Finally, we propose a composite reward mechanism to jointly optimize prediction accuracy and tool efficiency. Evaluations on both SWIG-HOI and HICO-DET datasets demonstrate that our method achieves state-of-the-art performance while requiring merely 36.7% of the training data compared to existing methods, validating our robustness, empirical effectiveness and efficiency. |
| title | What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2602.11499 |