Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.10739 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916362946871296 |
|---|---|
| author | Mai, Jinjie Zhu, Wenxuan Rojas, Sara Zarzar, Jesus Hamdi, Abdullah Qian, Guocheng Li, Bing Giancola, Silvio Ghanem, Bernard |
| author_facet | Mai, Jinjie Zhu, Wenxuan Rojas, Sara Zarzar, Jesus Hamdi, Abdullah Qian, Guocheng Li, Bing Giancola, Silvio Ghanem, Bernard |
| contents | Neural radiance fields (NeRFs) generally require many images with accurate poses for accurate novel view synthesis, which does not reflect realistic setups where views can be sparse and poses can be noisy. Previous solutions for learning NeRFs with sparse views and noisy poses only consider local geometry consistency with pairs of views. Closely following \textit{bundle adjustment} in Structure-from-Motion (SfM), we introduce TrackNeRF for more globally consistent geometry reconstruction and more accurate pose optimization. TrackNeRF introduces \textit{feature tracks}, \ie connected pixel trajectories across \textit{all} visible views that correspond to the \textit{same} 3D points. By enforcing reprojection consistency among feature tracks, TrackNeRF encourages holistic 3D consistency explicitly. Through extensive experiments, TrackNeRF sets a new benchmark in noisy and sparse view reconstruction. In particular, TrackNeRF shows significant improvements over the state-of-the-art BARF and SPARF by $\sim8$ and $\sim1$ in terms of PSNR on DTU under various sparse and noisy view setups. The code is available at \href{https://tracknerf.github.io/}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_10739 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TrackNeRF: Bundle Adjusting NeRF from Sparse and Noisy Views via Feature Tracks Mai, Jinjie Zhu, Wenxuan Rojas, Sara Zarzar, Jesus Hamdi, Abdullah Qian, Guocheng Li, Bing Giancola, Silvio Ghanem, Bernard Computer Vision and Pattern Recognition Neural radiance fields (NeRFs) generally require many images with accurate poses for accurate novel view synthesis, which does not reflect realistic setups where views can be sparse and poses can be noisy. Previous solutions for learning NeRFs with sparse views and noisy poses only consider local geometry consistency with pairs of views. Closely following \textit{bundle adjustment} in Structure-from-Motion (SfM), we introduce TrackNeRF for more globally consistent geometry reconstruction and more accurate pose optimization. TrackNeRF introduces \textit{feature tracks}, \ie connected pixel trajectories across \textit{all} visible views that correspond to the \textit{same} 3D points. By enforcing reprojection consistency among feature tracks, TrackNeRF encourages holistic 3D consistency explicitly. Through extensive experiments, TrackNeRF sets a new benchmark in noisy and sparse view reconstruction. In particular, TrackNeRF shows significant improvements over the state-of-the-art BARF and SPARF by $\sim8$ and $\sim1$ in terms of PSNR on DTU under various sparse and noisy view setups. The code is available at \href{https://tracknerf.github.io/}. |
| title | TrackNeRF: Bundle Adjusting NeRF from Sparse and Noisy Views via Feature Tracks |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.10739 |