Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916455803518976 |
|---|---|
| author | Terragni, Silvia Cuong, Hoang Daiber, Joachim Gudipati, Pallavi Mendes, Pablo N. |
| author_facet | Terragni, Silvia Cuong, Hoang Daiber, Joachim Gudipati, Pallavi Mendes, Pablo N. |
| contents | Large Language Models (LLMs) have demonstrated potential as effective search relevance evaluators. However, there is a lack of comprehensive guidance on which models consistently perform optimally across various contexts or within specific use cases. In this paper, we assess several LLMs and Multimodal Language Models (MLLMs) in terms of their alignment with human judgments across multiple multimodal search scenarios. Our analysis investigates the trade-offs between cost and accuracy, highlighting that model performance varies significantly depending on the context. Interestingly, in smaller models, the inclusion of a visual component may hinder performance rather than enhance it. These findings highlight the complexities involved in selecting the most appropriate model for practical applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_19974 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements Terragni, Silvia Cuong, Hoang Daiber, Joachim Gudipati, Pallavi Mendes, Pablo N. Machine Learning Computation and Language Information Retrieval Large Language Models (LLMs) have demonstrated potential as effective search relevance evaluators. However, there is a lack of comprehensive guidance on which models consistently perform optimally across various contexts or within specific use cases. In this paper, we assess several LLMs and Multimodal Language Models (MLLMs) in terms of their alignment with human judgments across multiple multimodal search scenarios. Our analysis investigates the trade-offs between cost and accuracy, highlighting that model performance varies significantly depending on the context. Interestingly, in smaller models, the inclusion of a visual component may hinder performance rather than enhance it. These findings highlight the complexities involved in selecting the most appropriate model for practical applications. |
| title | Evaluating Cost-Accuracy Trade-offs in Multimodal Search Relevance Judgements |
| topic | Machine Learning Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2410.19974 |