MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912243472400384 |
|---|---|
| author | Jiang, Hanwen Xu, Zexiang Xie, Desai Chen, Ziwen Jin, Haian Luan, Fujun Shu, Zhixin Zhang, Kai Bi, Sai Sun, Xin Gu, Jiuxiang Huang, Qixing Pavlakos, Georgios Tan, Hao |
| author_facet | Jiang, Hanwen Xu, Zexiang Xie, Desai Chen, Ziwen Jin, Haian Luan, Fujun Shu, Zhixin Zhang, Kai Bi, Sai Sun, Xin Gu, Jiuxiang Huang, Qixing Pavlakos, Georgios Tan, Hao |
| contents | We propose scaling up 3D scene reconstruction by training with synthesized data. At the core of our work is MegaSynth, a procedurally generated 3D dataset comprising 700K scenes - over 50 times larger than the prior real dataset DL3DV - dramatically scaling the training data. To enable scalable data generation, our key idea is eliminating semantic information, removing the need to model complex semantic priors such as object affordances and scene composition. Instead, we model scenes with basic spatial structures and geometry primitives, offering scalability. Besides, we control data complexity to facilitate training while loosely aligning it with real-world data distribution to benefit real-world generalization. We explore training LRMs with both MegaSynth and available real data. Experiment results show that joint training or pre-training with MegaSynth improves reconstruction quality by 1.2 to 1.8 dB PSNR across diverse image domains. Moreover, models trained solely on MegaSynth perform comparably to those trained on real data, underscoring the low-level nature of 3D reconstruction. Additionally, we provide an in-depth analysis of MegaSynth's properties for enhancing model capability, training stability, and generalization, as well as application to other tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_14166 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data Jiang, Hanwen Xu, Zexiang Xie, Desai Chen, Ziwen Jin, Haian Luan, Fujun Shu, Zhixin Zhang, Kai Bi, Sai Sun, Xin Gu, Jiuxiang Huang, Qixing Pavlakos, Georgios Tan, Hao Computer Vision and Pattern Recognition We propose scaling up 3D scene reconstruction by training with synthesized data. At the core of our work is MegaSynth, a procedurally generated 3D dataset comprising 700K scenes - over 50 times larger than the prior real dataset DL3DV - dramatically scaling the training data. To enable scalable data generation, our key idea is eliminating semantic information, removing the need to model complex semantic priors such as object affordances and scene composition. Instead, we model scenes with basic spatial structures and geometry primitives, offering scalability. Besides, we control data complexity to facilitate training while loosely aligning it with real-world data distribution to benefit real-world generalization. We explore training LRMs with both MegaSynth and available real data. Experiment results show that joint training or pre-training with MegaSynth improves reconstruction quality by 1.2 to 1.8 dB PSNR across diverse image domains. Moreover, models trained solely on MegaSynth perform comparably to those trained on real data, underscoring the low-level nature of 3D reconstruction. Additionally, we provide an in-depth analysis of MegaSynth's properties for enhancing model capability, training stability, and generalization, as well as application to other tasks. |
| title | MegaSynth: Scaling Up 3D Scene Reconstruction with Synthesized Data |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.14166 |