Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911730491195392 |
|---|---|
| author | Jang, Wonbong Liu, Shikun Sanyal, Soubhik Perez, Juan Camilo Ng, Kam Woh Agrawal, Sanskar Perez-Rua, Juan-Manuel Douratsos, Yiannis Xiang, Tao |
| author_facet | Jang, Wonbong Liu, Shikun Sanyal, Soubhik Perez, Juan Camilo Ng, Kam Woh Agrawal, Sanskar Perez-Rua, Juan-Manuel Douratsos, Yiannis Xiang, Tao |
| contents | Recovering camera parameters from images and rendering scenes from novel viewpoints have been treated as separate tasks in computer vision and graphics. This separation breaks down when image coverage is sparse or poses are ambiguous, since each task depends on what the other produces. We propose Rays as Pixels, a Video Diffusion Model (VDM) that learns a joint distribution over videos and camera trajectories. To our knowledge, this is the first model to predict camera poses and do camera-controlled video generation within a single framework. We represent each camera as dense ray pixels (raxels), a pixel-aligned encoding that lives in the same latent space as video frames, and denoise the two jointly through a Decoupled Self-Cross Attention mechanism. A single trained model handles three tasks: predicting camera trajectories from video, generating video from input images along a pre-defined trajectory, and jointly synthesizing video and trajectory from input images. We evaluate on pose estimation and camera-controlled video generation, and introduce a closed-loop self-consistency test showing that the model's predicted poses and its renderings conditioned on those poses agree. Ablations against Plücker embeddings confirm that representing cameras in a shared latent space with video is subtantially more effective. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_09429 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories Jang, Wonbong Liu, Shikun Sanyal, Soubhik Perez, Juan Camilo Ng, Kam Woh Agrawal, Sanskar Perez-Rua, Juan-Manuel Douratsos, Yiannis Xiang, Tao Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Recovering camera parameters from images and rendering scenes from novel viewpoints have been treated as separate tasks in computer vision and graphics. This separation breaks down when image coverage is sparse or poses are ambiguous, since each task depends on what the other produces. We propose Rays as Pixels, a Video Diffusion Model (VDM) that learns a joint distribution over videos and camera trajectories. To our knowledge, this is the first model to predict camera poses and do camera-controlled video generation within a single framework. We represent each camera as dense ray pixels (raxels), a pixel-aligned encoding that lives in the same latent space as video frames, and denoise the two jointly through a Decoupled Self-Cross Attention mechanism. A single trained model handles three tasks: predicting camera trajectories from video, generating video from input images along a pre-defined trajectory, and jointly synthesizing video and trajectory from input images. We evaluate on pose estimation and camera-controlled video generation, and introduce a closed-loop self-consistency test showing that the model's predicted poses and its renderings conditioned on those poses agree. Ablations against Plücker embeddings confirm that representing cameras in a shared latent space with video is subtantially more effective. |
| title | Rays as Pixels: Learning A Joint Distribution of Videos and Camera Trajectories |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2604.09429 |