Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.11350 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916901367578624 |
|---|---|
| author | Zhang, Zhenhao Wang, Hanqing Zeng, Xiangyu Cheng, Ziyu Liu, Jiaxin Yan, Haoyu Liu, Zhirui Ji, Kaiyang Gui, Tianxiang Hu, Ke Chen, Kangyi Fan, Yahao Pan, Mokai |
| author_facet | Zhang, Zhenhao Wang, Hanqing Zeng, Xiangyu Cheng, Ziyu Liu, Jiaxin Yan, Haoyu Liu, Zhirui Ji, Kaiyang Gui, Tianxiang Hu, Ke Chen, Kangyi Fan, Yahao Pan, Mokai |
| contents | Understanding and recognizing human-object interaction (HOI) is a pivotal application in AR/VR and robotics. Recent open-vocabulary HOI detection approaches depend exclusively on large language models for richer textual prompts, neglecting their inherent 3D spatial understanding capabilities. To address this shortcoming, we introduce HOID-R1, the first HOI detection framework that integrates chain-of-thought (CoT) guided supervised fine-tuning (SFT) with group relative policy optimization (GRPO) within a reinforcement learning (RL) paradigm. Specifically, we initially apply SFT to imbue the model with essential reasoning capabilities, forcing the model to articulate its thought process in the output. Subsequently, we integrate GRPO to leverage multi-reward signals for policy optimization, thereby enhancing alignment across diverse modalities. To mitigate hallucinations in the CoT reasoning, we introduce an "MLLM-as-a-judge" mechanism that supervises the CoT outputs, further improving generalization. Extensive experiments show that HOID-R1 achieves state-of-the-art performance on HOI detection benchmarks and outperforms existing methods in open-world generalization to novel scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_11350 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HOID-R1: Reinforcement Learning for Open-World Human-Object Interaction Detection Reasoning with Multimodal Large Language Model Zhang, Zhenhao Wang, Hanqing Zeng, Xiangyu Cheng, Ziyu Liu, Jiaxin Yan, Haoyu Liu, Zhirui Ji, Kaiyang Gui, Tianxiang Hu, Ke Chen, Kangyi Fan, Yahao Pan, Mokai Computer Vision and Pattern Recognition Understanding and recognizing human-object interaction (HOI) is a pivotal application in AR/VR and robotics. Recent open-vocabulary HOI detection approaches depend exclusively on large language models for richer textual prompts, neglecting their inherent 3D spatial understanding capabilities. To address this shortcoming, we introduce HOID-R1, the first HOI detection framework that integrates chain-of-thought (CoT) guided supervised fine-tuning (SFT) with group relative policy optimization (GRPO) within a reinforcement learning (RL) paradigm. Specifically, we initially apply SFT to imbue the model with essential reasoning capabilities, forcing the model to articulate its thought process in the output. Subsequently, we integrate GRPO to leverage multi-reward signals for policy optimization, thereby enhancing alignment across diverse modalities. To mitigate hallucinations in the CoT reasoning, we introduce an "MLLM-as-a-judge" mechanism that supervises the CoT outputs, further improving generalization. Extensive experiments show that HOID-R1 achieves state-of-the-art performance on HOI detection benchmarks and outperforms existing methods in open-world generalization to novel scenarios. |
| title | HOID-R1: Reinforcement Learning for Open-World Human-Object Interaction Detection Reasoning with Multimodal Large Language Model |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.11350 |