Connecting NeRFs, Images, and Text
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911836502228992 |
|---|---|
| author | Ballerini, Francesco Ramirez, Pierluigi Zama Mirabella, Roberto Salti, Samuele Di Stefano, Luigi |
| author_facet | Ballerini, Francesco Ramirez, Pierluigi Zama Mirabella, Roberto Salti, Samuele Di Stefano, Luigi |
| contents | Neural Radiance Fields (NeRFs) have emerged as a standard framework for representing 3D scenes and objects, introducing a novel data type for information exchange and storage. Concurrently, significant progress has been made in multimodal representation learning for text and image data. This paper explores a novel research direction that aims to connect the NeRF modality with other modalities, similar to established methodologies for images and text. To this end, we propose a simple framework that exploits pre-trained models for NeRF representations alongside multimodal models for text and image processing. Our framework learns a bidirectional mapping between NeRF embeddings and those obtained from corresponding images and text. This mapping unlocks several novel and useful applications, including NeRF zero-shot classification and NeRF retrieval from images or text. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_07993 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Connecting NeRFs, Images, and Text Ballerini, Francesco Ramirez, Pierluigi Zama Mirabella, Roberto Salti, Samuele Di Stefano, Luigi Computer Vision and Pattern Recognition Neural Radiance Fields (NeRFs) have emerged as a standard framework for representing 3D scenes and objects, introducing a novel data type for information exchange and storage. Concurrently, significant progress has been made in multimodal representation learning for text and image data. This paper explores a novel research direction that aims to connect the NeRF modality with other modalities, similar to established methodologies for images and text. To this end, we propose a simple framework that exploits pre-trained models for NeRF representations alongside multimodal models for text and image processing. Our framework learns a bidirectional mapping between NeRF embeddings and those obtained from corresponding images and text. This mapping unlocks several novel and useful applications, including NeRF zero-shot classification and NeRF retrieval from images or text. |
| title | Connecting NeRFs, Images, and Text |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.07993 |