ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908629455601664 |
|---|---|
| author | Sastry, Srikumar Khanal, Subash Dhakal, Aayush Lin, Jiayu Cher, Dan Jarosz, Phoenix Jacobs, Nathan |
| author_facet | Sastry, Srikumar Khanal, Subash Dhakal, Aayush Lin, Jiayu Cher, Dan Jarosz, Phoenix Jacobs, Nathan |
| contents | We introduce ProM3E, a probabilistic masked multimodal embedding model for any-to-any generation of multimodal representations for ecology. ProM3E is based on masked modality reconstruction in the embedding space, learning to infer missing modalities given a few context modalities. By design, our model supports modality inversion in the embedding space. The probabilistic nature of our model allows us to analyse the feasibility of fusing various modalities for given downstream tasks, essentially learning what to fuse. Using these features of our model, we propose a novel cross-modal retrieval approach that mixes inter-modal and intra-modal similarities to achieve superior performance across all retrieval tasks. We further leverage the hidden representation from our model to perform linear probing tasks and demonstrate the superior representation learning capability of our model. All our code, datasets and model will be released at https://vishu26.github.io/prom3e. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_02946 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology Sastry, Srikumar Khanal, Subash Dhakal, Aayush Lin, Jiayu Cher, Dan Jarosz, Phoenix Jacobs, Nathan Computer Vision and Pattern Recognition We introduce ProM3E, a probabilistic masked multimodal embedding model for any-to-any generation of multimodal representations for ecology. ProM3E is based on masked modality reconstruction in the embedding space, learning to infer missing modalities given a few context modalities. By design, our model supports modality inversion in the embedding space. The probabilistic nature of our model allows us to analyse the feasibility of fusing various modalities for given downstream tasks, essentially learning what to fuse. Using these features of our model, we propose a novel cross-modal retrieval approach that mixes inter-modal and intra-modal similarities to achieve superior performance across all retrieval tasks. We further leverage the hidden representation from our model to perform linear probing tasks and demonstrate the superior representation learning capability of our model. All our code, datasets and model will be released at https://vishu26.github.io/prom3e. |
| title | ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.02946 |