Scaling Sequence-to-Sequence Generative Neural Rendering
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909010864635904 |
|---|---|
| author | Liu, Shikun Ng, Kam Woh Jang, Wonbong Guo, Jiadong Han, Junlin Liu, Haozhe Douratsos, Yiannis Pérez, Juan C. Zhou, Zijian Phung, Chi Xiang, Tao Pérez-Rúa, Juan-Manuel |
| author_facet | Liu, Shikun Ng, Kam Woh Jang, Wonbong Guo, Jiadong Han, Junlin Liu, Haozhe Douratsos, Yiannis Pérez, Juan C. Zhou, Zijian Phung, Chi Xiang, Tao Pérez-Rúa, Juan-Manuel |
| contents | We present Kaleido, a family of generative models designed for photorealistic, unified object- and scene-level neural rendering. Kaleido operates on the principle that 3D can be regarded as a specialised sub-domain of video, expressed purely as a sequence-to-sequence image synthesis task. Through a systemic study of scaling sequence-to-sequence generative neural rendering, we introduce key architectural innovations that enable our model to: i) perform generative view synthesis without explicit 3D representations; ii) generate any number of 6-DoF target views conditioned on any number of reference views via a masked autoregressive framework; and iii) seamlessly unify 3D and video modelling within a single decoder-only rectified flow transformer. Within this unified framework, Kaleido leverages large-scale video data for pre-training, which significantly improves spatial consistency and reduces reliance on scarce, camera-labelled 3D datasets -- all without any architectural modifications. Kaleido sets a new state-of-the-art on a range of view synthesis benchmarks. Its zero-shot performance substantially outperforms other generative methods in few-view settings, and, for the first time, matches the quality of per-scene optimisation methods in many-view settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04236 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Sequence-to-Sequence Generative Neural Rendering Liu, Shikun Ng, Kam Woh Jang, Wonbong Guo, Jiadong Han, Junlin Liu, Haozhe Douratsos, Yiannis Pérez, Juan C. Zhou, Zijian Phung, Chi Xiang, Tao Pérez-Rúa, Juan-Manuel Computer Vision and Pattern Recognition We present Kaleido, a family of generative models designed for photorealistic, unified object- and scene-level neural rendering. Kaleido operates on the principle that 3D can be regarded as a specialised sub-domain of video, expressed purely as a sequence-to-sequence image synthesis task. Through a systemic study of scaling sequence-to-sequence generative neural rendering, we introduce key architectural innovations that enable our model to: i) perform generative view synthesis without explicit 3D representations; ii) generate any number of 6-DoF target views conditioned on any number of reference views via a masked autoregressive framework; and iii) seamlessly unify 3D and video modelling within a single decoder-only rectified flow transformer. Within this unified framework, Kaleido leverages large-scale video data for pre-training, which significantly improves spatial consistency and reduces reliance on scarce, camera-labelled 3D datasets -- all without any architectural modifications. Kaleido sets a new state-of-the-art on a range of view synthesis benchmarks. Its zero-shot performance substantially outperforms other generative methods in few-view settings, and, for the first time, matches the quality of per-scene optimisation methods in many-view settings. |
| title | Scaling Sequence-to-Sequence Generative Neural Rendering |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.04236 |