HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912205136461824 |
|---|---|
| author | Zhao, Jiaxing Yang, Qize Peng, Yixing Bai, Detao Yao, Shimin Sun, Boyuan Chen, Xiang Fu, Shenghao chen, Weixuan Wei, Xihan Bo, Liefeng |
| author_facet | Zhao, Jiaxing Yang, Qize Peng, Yixing Bai, Detao Yao, Shimin Sun, Boyuan Chen, Xiang Fu, Shenghao chen, Weixuan Wei, Xihan Bo, Liefeng |
| contents | In human-centric scenes, the ability to simultaneously understand visual and auditory information is crucial. While recent omni models can process multiple modalities, they generally lack effectiveness in human-centric scenes due to the absence of large-scale, specialized datasets and non-targeted architectures. In this work, we developed HumanOmni, the industry's first human-centric Omni-multimodal large language model. We constructed a dataset containing over 2.4 million human-centric video clips with detailed captions and more than 14 million instructions, facilitating the understanding of diverse human-centric scenes. HumanOmni includes three specialized branches for understanding different types of scenes. It adaptively fuses features from these branches based on user instructions, significantly enhancing visual understanding in scenes centered around individuals. Moreover, HumanOmni integrates audio features to ensure a comprehensive understanding of environments and individuals. Our experiments validate HumanOmni's advanced capabilities in handling human-centric scenes across a variety of tasks, including emotion recognition, facial expression description, and action understanding. Our model will be open-sourced to facilitate further development and collaboration within both academia and industry. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_15111 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding Zhao, Jiaxing Yang, Qize Peng, Yixing Bai, Detao Yao, Shimin Sun, Boyuan Chen, Xiang Fu, Shenghao chen, Weixuan Wei, Xihan Bo, Liefeng Computer Vision and Pattern Recognition In human-centric scenes, the ability to simultaneously understand visual and auditory information is crucial. While recent omni models can process multiple modalities, they generally lack effectiveness in human-centric scenes due to the absence of large-scale, specialized datasets and non-targeted architectures. In this work, we developed HumanOmni, the industry's first human-centric Omni-multimodal large language model. We constructed a dataset containing over 2.4 million human-centric video clips with detailed captions and more than 14 million instructions, facilitating the understanding of diverse human-centric scenes. HumanOmni includes three specialized branches for understanding different types of scenes. It adaptively fuses features from these branches based on user instructions, significantly enhancing visual understanding in scenes centered around individuals. Moreover, HumanOmni integrates audio features to ensure a comprehensive understanding of environments and individuals. Our experiments validate HumanOmni's advanced capabilities in handling human-centric scenes across a variety of tasks, including emotion recognition, facial expression description, and action understanding. Our model will be open-sourced to facilitate further development and collaboration within both academia and industry. |
| title | HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.15111 |