From Latent to Engine Manifolds: Analyzing ImageBind's Multimodal Embedding Space
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914950520242176 |
|---|---|
| author | Hamara, Andrew Rivas, Pablo |
| author_facet | Hamara, Andrew Rivas, Pablo |
| contents | This study investigates ImageBind's ability to generate meaningful fused multimodal embeddings for online auto parts listings. We propose a simplistic embedding fusion workflow that aims to capture the overlapping information of image/text pairs, ultimately combining the semantics of a post into a joint embedding. After storing such fused embeddings in a vector database, we experiment with dimensionality reduction and provide empirical evidence to convey the semantic quality of the joint embeddings by clustering and examining the posts nearest to each cluster centroid. Additionally, our initial findings with ImageBind's emergent zero-shot cross-modal retrieval suggest that pure audio embeddings can correlate with semantically similar marketplace listings, indicating potential avenues for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10528 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Latent to Engine Manifolds: Analyzing ImageBind's Multimodal Embedding Space Hamara, Andrew Rivas, Pablo Computer Vision and Pattern Recognition Artificial Intelligence I.2.10; I.5.4 This study investigates ImageBind's ability to generate meaningful fused multimodal embeddings for online auto parts listings. We propose a simplistic embedding fusion workflow that aims to capture the overlapping information of image/text pairs, ultimately combining the semantics of a post into a joint embedding. After storing such fused embeddings in a vector database, we experiment with dimensionality reduction and provide empirical evidence to convey the semantic quality of the joint embeddings by clustering and examining the posts nearest to each cluster centroid. Additionally, our initial findings with ImageBind's emergent zero-shot cross-modal retrieval suggest that pure audio embeddings can correlate with semantically similar marketplace listings, indicating potential avenues for future research. |
| title | From Latent to Engine Manifolds: Analyzing ImageBind's Multimodal Embedding Space |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence I.2.10; I.5.4 |
| url | https://arxiv.org/abs/2409.10528 |