Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.14462 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909707223957504 |
|---|---|
| author | Marrie, Juliette Menegaux, Romain Arbel, Michael Larlus, Diane Mairal, Julien |
| author_facet | Marrie, Juliette Menegaux, Romain Arbel, Michael Larlus, Diane Mairal, Julien |
| contents | We address the problem of extending the capabilities of vision foundation models such as DINO, SAM, and CLIP, to 3D tasks. Specifically, we introduce a novel method to uplift 2D image features into Gaussian Splatting representations of 3D scenes. Unlike traditional approaches that rely on minimizing a reconstruction loss, our method employs a simpler and more efficient feature aggregation technique, augmented by a graph diffusion mechanism. Graph diffusion refines 3D features, such as coarse segmentation masks, by leveraging 3D geometry and pairwise similarities induced by DINOv2. Our approach achieves performance comparable to the state of the art on multiple downstream tasks while delivering significant speed-ups. Notably, we obtain competitive segmentation results using only generic DINOv2 features, despite DINOv2 not being trained on millions of annotated segmentation masks like SAM. When applied to CLIP features, our method demonstrates strong performance in open-vocabulary object segmentation tasks, highlighting the versatility of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14462 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenes Marrie, Juliette Menegaux, Romain Arbel, Michael Larlus, Diane Mairal, Julien Computer Vision and Pattern Recognition We address the problem of extending the capabilities of vision foundation models such as DINO, SAM, and CLIP, to 3D tasks. Specifically, we introduce a novel method to uplift 2D image features into Gaussian Splatting representations of 3D scenes. Unlike traditional approaches that rely on minimizing a reconstruction loss, our method employs a simpler and more efficient feature aggregation technique, augmented by a graph diffusion mechanism. Graph diffusion refines 3D features, such as coarse segmentation masks, by leveraging 3D geometry and pairwise similarities induced by DINOv2. Our approach achieves performance comparable to the state of the art on multiple downstream tasks while delivering significant speed-ups. Notably, we obtain competitive segmentation results using only generic DINOv2 features, despite DINOv2 not being trained on millions of annotated segmentation masks like SAM. When applied to CLIP features, our method demonstrates strong performance in open-vocabulary object segmentation tasks, highlighting the versatility of our approach. |
| title | LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenes |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.14462 |