Scaling Transformer-Based Novel View Synthesis Models with Token Disentanglement and Synthetic Data
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911143027539968 |
|---|---|
| author | Nair, Nithin Gopalakrishnan Kaza, Srinivas Luo, Xuan Patel, Vishal M. Lombardi, Stephen Park, Jungyeon |
| author_facet | Nair, Nithin Gopalakrishnan Kaza, Srinivas Luo, Xuan Patel, Vishal M. Lombardi, Stephen Park, Jungyeon |
| contents | Large transformer-based models have made significant progress in generalizable novel view synthesis (NVS) from sparse input views, generating novel viewpoints without the need for test-time optimization. However, these models are constrained by the limited diversity of publicly available scene datasets, making most real-world (in-the-wild) scenes out-of-distribution. To overcome this, we incorporate synthetic training data generated from diffusion models, which improves generalization across unseen domains. While synthetic data offers scalability, we identify artifacts introduced during data generation as a key bottleneck affecting reconstruction quality. To address this, we propose a token disentanglement process within the transformer architecture, enhancing feature separation and ensuring more effective learning. This refinement not only improves reconstruction quality over standard transformers but also enables scalable training with synthetic data. As a result, our method outperforms existing models on both in-dataset and cross-dataset evaluations, achieving state-of-the-art results across multiple benchmarks while significantly reducing computational costs. Project page: https://scaling3dnvs.github.io/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_06950 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Transformer-Based Novel View Synthesis Models with Token Disentanglement and Synthetic Data Nair, Nithin Gopalakrishnan Kaza, Srinivas Luo, Xuan Patel, Vishal M. Lombardi, Stephen Park, Jungyeon Graphics Computer Vision and Pattern Recognition Large transformer-based models have made significant progress in generalizable novel view synthesis (NVS) from sparse input views, generating novel viewpoints without the need for test-time optimization. However, these models are constrained by the limited diversity of publicly available scene datasets, making most real-world (in-the-wild) scenes out-of-distribution. To overcome this, we incorporate synthetic training data generated from diffusion models, which improves generalization across unseen domains. While synthetic data offers scalability, we identify artifacts introduced during data generation as a key bottleneck affecting reconstruction quality. To address this, we propose a token disentanglement process within the transformer architecture, enhancing feature separation and ensuring more effective learning. This refinement not only improves reconstruction quality over standard transformers but also enables scalable training with synthetic data. As a result, our method outperforms existing models on both in-dataset and cross-dataset evaluations, achieving state-of-the-art results across multiple benchmarks while significantly reducing computational costs. Project page: https://scaling3dnvs.github.io/ |
| title | Scaling Transformer-Based Novel View Synthesis Models with Token Disentanglement and Synthetic Data |
| topic | Graphics Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.06950 |