LoLep: Single-View View Synthesis with Locally-Learned Planes and Self-Attention Occlusion Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911455618531328 |
|---|---|
| author | Wang, Cong Wang, Yu-Ping Manocha, Dinesh |
| author_facet | Wang, Cong Wang, Yu-Ping Manocha, Dinesh |
| contents | We propose a novel method, LoLep, which regresses Locally-Learned planes from a single RGB image to represent scenes accurately, thus generating better novel views. Without the depth information, regressing appropriate plane locations is a challenging problem. To solve this issue, we pre-partition the disparity space into bins and design a disparity sampler to regress local offsets for multiple planes in each bin. However, only using such a sampler makes the network not convergent; we further propose two optimizing strategies that combine with different disparity distributions of datasets and propose an occlusion-aware reprojection loss as a simple yet effective geometric supervision technique. We also introduce a self-attention mechanism to improve occlusion inference and present a Block-Sampling Self-Attention (BS-SA) module to address the problem of applying self-attention to large feature maps. We demonstrate the effectiveness of our approach and generate state-of-the-art results on different datasets. Compared to MINE, our approach has an LPIPS reduction of 4.8%-9.0% and an RV reduction of 73.9%-83.5%. We also evaluate the performance on real-world images and demonstrate the benefits. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_12217 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | LoLep: Single-View View Synthesis with Locally-Learned Planes and Self-Attention Occlusion Inference Wang, Cong Wang, Yu-Ping Manocha, Dinesh Computer Vision and Pattern Recognition We propose a novel method, LoLep, which regresses Locally-Learned planes from a single RGB image to represent scenes accurately, thus generating better novel views. Without the depth information, regressing appropriate plane locations is a challenging problem. To solve this issue, we pre-partition the disparity space into bins and design a disparity sampler to regress local offsets for multiple planes in each bin. However, only using such a sampler makes the network not convergent; we further propose two optimizing strategies that combine with different disparity distributions of datasets and propose an occlusion-aware reprojection loss as a simple yet effective geometric supervision technique. We also introduce a self-attention mechanism to improve occlusion inference and present a Block-Sampling Self-Attention (BS-SA) module to address the problem of applying self-attention to large feature maps. We demonstrate the effectiveness of our approach and generate state-of-the-art results on different datasets. Compared to MINE, our approach has an LPIPS reduction of 4.8%-9.0% and an RV reduction of 73.9%-83.5%. We also evaluate the performance on real-world images and demonstrate the benefits. |
| title | LoLep: Single-View View Synthesis with Locally-Learned Planes and Self-Attention Occlusion Inference |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2307.12217 |