Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914216663842816 |
|---|---|
| author | Zhu, Tianrui Zhang, Shiyi Sun, Zhirui Tian, Jingqi Tang, Yansong |
| author_facet | Zhu, Tianrui Zhang, Shiyi Sun, Zhirui Tian, Jingqi Tang, Yansong |
| contents | Frame-level autoregressive (frame-AR) models have achieved significant progress, enabling real-time video generation comparable to bidirectional diffusion models and serving as a foundation for interactive world models and game engines. However, current approaches in long video generation typically rely on window attention, which naively discards historical context outside the window, leading to catastrophic forgetting and scene inconsistency; conversely, retaining full history incurs prohibitive memory costs. To address this trade-off, we propose Memorize-and-Generate (MAG), a framework that decouples memory compression and frame generation into distinct tasks. Specifically, we train a memory model to compress historical information into a compact KV cache, and a separate generator model to synthesize subsequent frames utilizing this compressed representation. Furthermore, we introduce MAG-Bench to strictly evaluate historical memory retention. Extensive experiments demonstrate that MAG achieves superior historical scene consistency while maintaining competitive performance on standard video generation benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_18741 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generation Zhu, Tianrui Zhang, Shiyi Sun, Zhirui Tian, Jingqi Tang, Yansong Computer Vision and Pattern Recognition Frame-level autoregressive (frame-AR) models have achieved significant progress, enabling real-time video generation comparable to bidirectional diffusion models and serving as a foundation for interactive world models and game engines. However, current approaches in long video generation typically rely on window attention, which naively discards historical context outside the window, leading to catastrophic forgetting and scene inconsistency; conversely, retaining full history incurs prohibitive memory costs. To address this trade-off, we propose Memorize-and-Generate (MAG), a framework that decouples memory compression and frame generation into distinct tasks. Specifically, we train a memory model to compress historical information into a compact KV cache, and a separate generator model to synthesize subsequent frames utilizing this compressed representation. Furthermore, we introduce MAG-Bench to strictly evaluate historical memory retention. Extensive experiments demonstrate that MAG achieves superior historical scene consistency while maintaining competitive performance on standard video generation benchmarks. |
| title | Memorize-and-Generate: Towards Long-Term Consistency in Real-Time Video Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.18741 |