A Study of Failure Modes in Two-Stage Human-Object Interaction Detection
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910130245730304 |
|---|---|
| author | Wang, Lemeng Lei, Qinqian Bakshi, Vidhi Yi, Daniel Liu, Yifan Hou, Jiacheng Hao, Asher Seng Mai, Zheda Chao, Wei-Lun Tan, Robby T. Wang, Bo |
| author_facet | Wang, Lemeng Lei, Qinqian Bakshi, Vidhi Yi, Daniel Liu, Yifan Hou, Jiacheng Hao, Asher Seng Mai, Zheda Chao, Wei-Lun Tan, Robby T. Wang, Bo |
| contents | Human-object interaction (HOI) detection aims to detect interactions between humans and objects in images. While recent advances have improved performance on existing benchmarks, their evaluations mainly focus on overall prediction accuracy and provide limited insight into the underlying causes of model failures. In particular, modern models often struggle in complex scenes involving multiple people and rare interaction combinations. In this work, we present a study to better understand the failure modes of two-stage HOI models, which form the basis of many current HOI detection approaches. Rather than constructing a large-scale benchmark, we instead decompose HOI detection into multiple interpretable perspectives and analyze model behavior across these dimensions to study different types of failure patterns. We curate a subset of images from an existing HOI dataset organized by human-object-interaction configurations (e.g., multi-person interactions and object sharing), and analyze model behavior under these configurations to examine different failure modes. This design allows us to analyze how these HOI models behave under different scene compositions and why their predictions fail. Importantly, high overall benchmark performance does not necessarily reflect robust visual reasoning about human-object relationships. We hope that this study can provide useful insights into the limitations of HOI models and offer observations for future research in this area. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_13448 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | A Study of Failure Modes in Two-Stage Human-Object Interaction Detection Wang, Lemeng Lei, Qinqian Bakshi, Vidhi Yi, Daniel Liu, Yifan Hou, Jiacheng Hao, Asher Seng Mai, Zheda Chao, Wei-Lun Tan, Robby T. Wang, Bo Computer Vision and Pattern Recognition Artificial Intelligence Human-object interaction (HOI) detection aims to detect interactions between humans and objects in images. While recent advances have improved performance on existing benchmarks, their evaluations mainly focus on overall prediction accuracy and provide limited insight into the underlying causes of model failures. In particular, modern models often struggle in complex scenes involving multiple people and rare interaction combinations. In this work, we present a study to better understand the failure modes of two-stage HOI models, which form the basis of many current HOI detection approaches. Rather than constructing a large-scale benchmark, we instead decompose HOI detection into multiple interpretable perspectives and analyze model behavior across these dimensions to study different types of failure patterns. We curate a subset of images from an existing HOI dataset organized by human-object-interaction configurations (e.g., multi-person interactions and object sharing), and analyze model behavior under these configurations to examine different failure modes. This design allows us to analyze how these HOI models behave under different scene compositions and why their predictions fail. Importantly, high overall benchmark performance does not necessarily reflect robust visual reasoning about human-object relationships. We hope that this study can provide useful insights into the limitations of HOI models and offer observations for future research in this area. |
| title | A Study of Failure Modes in Two-Stage Human-Object Interaction Detection |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2604.13448 |