OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913018658422784 |
|---|---|
| author | Jia, Yiduo Zhu, Muzhi Zhong, Hao Liu, Mingyu Xi, Yuling Chen, Hao Qin, Bin Yang, Yongjie Luo, Zhenbo Shen, Chunhua |
| author_facet | Jia, Yiduo Zhu, Muzhi Zhong, Hao Liu, Mingyu Xi, Yuling Chen, Hao Qin, Bin Yang, Yongjie Luo, Zhenbo Shen, Chunhua |
| contents | To extend the reinforcement learning post-training paradigm to omni-modal models for concurrently bolstering video-audio understanding and collaborative reasoning, we propose OmniJigsaw, a generic self-supervised framework built upon a temporal reordering proxy task. Centered on the chronological reconstruction of shuffled audio-visual clips, this paradigm strategically orchestrates visual and auditory signals to compel cross-modal integration through three distinct strategies: Joint Modality Integration, Sample-level Modality Selection, and Clip-level Modality Masking. Recognizing that the efficacy of such proxy tasks is fundamentally tied to puzzle quality, we design a two-stage coarse-to-fine data filtering pipeline, which facilitates the efficient adaptation of OmniJigsaw to massive unannotated omni-modal data. Our analysis reveals a ``bi-modal shortcut phenomenon'' in joint modality integration and demonstrates that fine-grained clip-level modality masking mitigates this issue while outperforming sample-level modality selection. Extensive evaluations on 15 benchmarks show substantial gains in video, audio, and collaborative reasoning, validating OmniJigsaw as a scalable paradigm for self-supervised omni-modal learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08209 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering Jia, Yiduo Zhu, Muzhi Zhong, Hao Liu, Mingyu Xi, Yuling Chen, Hao Qin, Bin Yang, Yongjie Luo, Zhenbo Shen, Chunhua Computer Vision and Pattern Recognition To extend the reinforcement learning post-training paradigm to omni-modal models for concurrently bolstering video-audio understanding and collaborative reasoning, we propose OmniJigsaw, a generic self-supervised framework built upon a temporal reordering proxy task. Centered on the chronological reconstruction of shuffled audio-visual clips, this paradigm strategically orchestrates visual and auditory signals to compel cross-modal integration through three distinct strategies: Joint Modality Integration, Sample-level Modality Selection, and Clip-level Modality Masking. Recognizing that the efficacy of such proxy tasks is fundamentally tied to puzzle quality, we design a two-stage coarse-to-fine data filtering pipeline, which facilitates the efficient adaptation of OmniJigsaw to massive unannotated omni-modal data. Our analysis reveals a ``bi-modal shortcut phenomenon'' in joint modality integration and demonstrates that fine-grained clip-level modality masking mitigates this issue while outperforming sample-level modality selection. Extensive evaluations on 15 benchmarks show substantial gains in video, audio, and collaborative reasoning, validating OmniJigsaw as a scalable paradigm for self-supervised omni-modal learning. |
| title | OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2604.08209 |