Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.18901 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912445141876736 |
|---|---|
| author | Sun, Wenqiang Wei, Fangyun Zhao, Jinjing Chen, Xi Chen, Zilong Zhang, Hongyang Zhang, Jun Lu, Yan |
| author_facet | Sun, Wenqiang Wei, Fangyun Zhao, Jinjing Chen, Xi Chen, Zilong Zhang, Hongyang Zhang, Jun Lu, Yan |
| contents | We introduce RealPlay, a neural network-based real-world game engine that enables interactive video generation from user control signals. Unlike prior works focused on game-style visuals, RealPlay aims to produce photorealistic, temporally consistent video sequences that resemble real-world footage. It operates in an interactive loop: users observe a generated scene, issue a control command, and receive a short video chunk in response. To enable such realistic and responsive generation, we address key challenges including iterative chunk-wise prediction for low-latency feedback, temporal consistency across iterations, and accurate control response. RealPlay is trained on a combination of labeled game data and unlabeled real-world videos, without requiring real-world action annotations. Notably, we observe two forms of generalization: (1) control transfer-RealPlay effectively maps control signals from virtual to real-world scenarios; and (2) entity transfer-although training labels originate solely from a car racing game, RealPlay generalizes to control diverse real-world entities, including bicycles and pedestrians, beyond vehicles. Project page can be found: https://wenqsun.github.io/RealPlay/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_18901 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Virtual Games to Real-World Play Sun, Wenqiang Wei, Fangyun Zhao, Jinjing Chen, Xi Chen, Zilong Zhang, Hongyang Zhang, Jun Lu, Yan Computer Vision and Pattern Recognition We introduce RealPlay, a neural network-based real-world game engine that enables interactive video generation from user control signals. Unlike prior works focused on game-style visuals, RealPlay aims to produce photorealistic, temporally consistent video sequences that resemble real-world footage. It operates in an interactive loop: users observe a generated scene, issue a control command, and receive a short video chunk in response. To enable such realistic and responsive generation, we address key challenges including iterative chunk-wise prediction for low-latency feedback, temporal consistency across iterations, and accurate control response. RealPlay is trained on a combination of labeled game data and unlabeled real-world videos, without requiring real-world action annotations. Notably, we observe two forms of generalization: (1) control transfer-RealPlay effectively maps control signals from virtual to real-world scenarios; and (2) entity transfer-although training labels originate solely from a car racing game, RealPlay generalizes to control diverse real-world entities, including bicycles and pedestrians, beyond vehicles. Project page can be found: https://wenqsun.github.io/RealPlay/ |
| title | From Virtual Games to Real-World Play |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.18901 |