Self-Aligning Depth-regularized Radiance Fields for Asynchronous RGB-D Sequences
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2022
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910397493149696 |
|---|---|
| author | Huang, Yuxin Yang, Andong Wu, Zirui Chen, Yuantao Yang, Runyi Zhu, Zhenxin Hou, Chao Zhao, Hao Zhou, Guyue |
| author_facet | Huang, Yuxin Yang, Andong Wu, Zirui Chen, Yuantao Yang, Runyi Zhu, Zhenxin Hou, Chao Zhao, Hao Zhou, Guyue |
| contents | It has been shown that learning radiance fields with depth rendering and depth supervision can effectively promote the quality and convergence of view synthesis. However, this paradigm requires input RGB-D sequences to be synchronized, hindering its usage in the UAV city modeling scenario. As there exists asynchrony between RGB images and depth images due to high-speed flight, we propose a novel time-pose function, which is an implicit network that maps timestamps to $\rm SE(3)$ elements. To simplify the training process, we also design a joint optimization scheme to jointly learn the large-scale depth-regularized radiance fields and the time-pose function. Our algorithm consists of three steps: (1) time-pose function fitting, (2) radiance field bootstrapping, (3) joint pose error compensation and radiance field refinement. In addition, we propose a large synthetic dataset with diverse controlled mismatches and ground truth to evaluate this new problem setting systematically. Through extensive experiments, we demonstrate that our method outperforms baselines without regularization. We also show qualitatively improved results on a real-world asynchronous RGB-D sequence captured by drone. Codes, data, and models will be made publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2211_07459 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Self-Aligning Depth-regularized Radiance Fields for Asynchronous RGB-D Sequences Huang, Yuxin Yang, Andong Wu, Zirui Chen, Yuantao Yang, Runyi Zhu, Zhenxin Hou, Chao Zhao, Hao Zhou, Guyue Computer Vision and Pattern Recognition Robotics It has been shown that learning radiance fields with depth rendering and depth supervision can effectively promote the quality and convergence of view synthesis. However, this paradigm requires input RGB-D sequences to be synchronized, hindering its usage in the UAV city modeling scenario. As there exists asynchrony between RGB images and depth images due to high-speed flight, we propose a novel time-pose function, which is an implicit network that maps timestamps to $\rm SE(3)$ elements. To simplify the training process, we also design a joint optimization scheme to jointly learn the large-scale depth-regularized radiance fields and the time-pose function. Our algorithm consists of three steps: (1) time-pose function fitting, (2) radiance field bootstrapping, (3) joint pose error compensation and radiance field refinement. In addition, we propose a large synthetic dataset with diverse controlled mismatches and ground truth to evaluate this new problem setting systematically. Through extensive experiments, we demonstrate that our method outperforms baselines without regularization. We also show qualitatively improved results on a real-world asynchronous RGB-D sequence captured by drone. Codes, data, and models will be made publicly available. |
| title | Self-Aligning Depth-regularized Radiance Fields for Asynchronous RGB-D Sequences |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2211.07459 |