Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.22905 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911349755346944 |
|---|---|
| author | Liu, Kai Li, Jungang Sun, Yuchong Wu, Shengqiong Gao, Jianzhang Zhang, Daoan Zhang, Wei Jin, Sheng Yu, Sicheng Zhan, Geng Ji, Jiayi Zhou, Fan Zheng, Liang Yan, Shuicheng Fei, Hao Chua, Tat-Seng |
| author_facet | Liu, Kai Li, Jungang Sun, Yuchong Wu, Shengqiong Gao, Jianzhang Zhang, Daoan Zhang, Wei Jin, Sheng Yu, Sicheng Zhan, Geng Ji, Jiayi Zhou, Fan Zheng, Liang Yan, Shuicheng Fei, Hao Chua, Tat-Seng |
| contents | This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for joint audio-video (JAV) comprehension and generation. JavisGPT has a concise encoder-LLM-decoder architecture, which has a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. For instruction tuning, we construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that cover diverse and multi-level comprehension and generation scenarios. On JAV comprehension and generation benchmarks, our experiments show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_22905 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation Liu, Kai Li, Jungang Sun, Yuchong Wu, Shengqiong Gao, Jianzhang Zhang, Daoan Zhang, Wei Jin, Sheng Yu, Sicheng Zhan, Geng Ji, Jiayi Zhou, Fan Zheng, Liang Yan, Shuicheng Fei, Hao Chua, Tat-Seng Computer Vision and Pattern Recognition This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for joint audio-video (JAV) comprehension and generation. JavisGPT has a concise encoder-LLM-decoder architecture, which has a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. For instruction tuning, we construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that cover diverse and multi-level comprehension and generation scenarios. On JAV comprehension and generation benchmarks, our experiments show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings. |
| title | JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.22905 |