Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.08995 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917401070665728 |
|---|---|
| author | Wang, Zile Liu, Zexiang Li, Jiaxing Huang, Kaichen Xu, Baixin Kang, Fei An, Mengyin Wang, Peiyu Jiang, Biao Wei, Yichen Xietian, Yidan Pei, Jiangbo Hu, Liang Jiang, Boyi Xue, Hua Wang, Zidong Sun, Haofeng Li, Wei Ouyang, Wanli He, Xianglong Liu, Yang Li, Yangguang Zhou, Yahui |
| author_facet | Wang, Zile Liu, Zexiang Li, Jiaxing Huang, Kaichen Xu, Baixin Kang, Fei An, Mengyin Wang, Peiyu Jiang, Biao Wei, Yichen Xietian, Yidan Pei, Jiangbo Hu, Liang Jiang, Boyi Xue, Hua Wang, Zidong Sun, Haofeng Li, Wei Ouyang, Wanli He, Xianglong Liu, Yang Li, Yangguang Zhou, Yahui |
| contents | With the advancement of interactive video generation, diffusion models have increasingly demonstrated their potential as world models. However, existing approaches still struggle to simultaneously achieve memory-enabled long-term temporal consistency and high-resolution real-time generation, limiting their applicability in real-world scenarios. To address this, we present Matrix-Game 3.0, a memory-augmented interactive world model designed for 720p real-time longform video generation. Building upon Matrix-Game 2.0, we introduce systematic improvements across data, model, and inference. First, we develop an upgraded industrial-scale infinite data engine that integrates Unreal Engine-based synthetic data, large-scale automated collection from AAA games, and real-world video augmentation to produce high-quality Video-Pose-Action-Prompt quadruplet data at scale. Second, we propose a training framework for long-horizon consistency: by modeling prediction residuals and re-injecting imperfect generated frames during training, the base model learns self-correction; meanwhile, camera-aware memory retrieval and injection enable the base model to achieve long horizon spatiotemporal consistency. Third, we design a multi-segment autoregressive distillation strategy based on Distribution Matching Distillation (DMD), combined with model quantization and VAE decoder pruning, to achieve efficient real-time inference. Experimental results show that Matrix-Game 3.0 achieves up to 40 FPS real-time generation at 720p resolution with a 5B model, while maintaining stable memory consistency over minute-long sequences. Scaling up to a 2x14B model further improves generation quality, dynamics, and generalization. Our approach provides a practical pathway toward industrial-scale deployable world models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08995 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory Wang, Zile Liu, Zexiang Li, Jiaxing Huang, Kaichen Xu, Baixin Kang, Fei An, Mengyin Wang, Peiyu Jiang, Biao Wei, Yichen Xietian, Yidan Pei, Jiangbo Hu, Liang Jiang, Boyi Xue, Hua Wang, Zidong Sun, Haofeng Li, Wei Ouyang, Wanli He, Xianglong Liu, Yang Li, Yangguang Zhou, Yahui Computer Vision and Pattern Recognition With the advancement of interactive video generation, diffusion models have increasingly demonstrated their potential as world models. However, existing approaches still struggle to simultaneously achieve memory-enabled long-term temporal consistency and high-resolution real-time generation, limiting their applicability in real-world scenarios. To address this, we present Matrix-Game 3.0, a memory-augmented interactive world model designed for 720p real-time longform video generation. Building upon Matrix-Game 2.0, we introduce systematic improvements across data, model, and inference. First, we develop an upgraded industrial-scale infinite data engine that integrates Unreal Engine-based synthetic data, large-scale automated collection from AAA games, and real-world video augmentation to produce high-quality Video-Pose-Action-Prompt quadruplet data at scale. Second, we propose a training framework for long-horizon consistency: by modeling prediction residuals and re-injecting imperfect generated frames during training, the base model learns self-correction; meanwhile, camera-aware memory retrieval and injection enable the base model to achieve long horizon spatiotemporal consistency. Third, we design a multi-segment autoregressive distillation strategy based on Distribution Matching Distillation (DMD), combined with model quantization and VAE decoder pruning, to achieve efficient real-time inference. Experimental results show that Matrix-Game 3.0 achieves up to 40 FPS real-time generation at 720p resolution with a 5B model, while maintaining stable memory consistency over minute-long sequences. Scaling up to a 2x14B model further improves generation quality, dynamics, and generalization. Our approach provides a practical pathway toward industrial-scale deployable world models. |
| title | Matrix-Game 3.0: Real-Time and Streaming Interactive World Model with Long-Horizon Memory |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2604.08995 |