Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2411.12287 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917964241960960 |
|---|---|
| author | Go, Dongyoung Whang, Taesun Lee, Chanhee Kim, Hwa-Yeon Park, Sunghoon Ji, Seunghwan Kim, Jinho Kim, Dongchan Kim, Young-Bum |
| author_facet | Go, Dongyoung Whang, Taesun Lee, Chanhee Kim, Hwa-Yeon Park, Sunghoon Ji, Seunghwan Kim, Jinho Kim, Dongchan Kim, Young-Bum |
| contents | The integration of Retrieval-Augmented Generation (RAG) with Multimodal Large Language Models (MLLMs) has revolutionized information retrieval and expanded the practical applications of AI. However, current systems struggle in accurately interpreting user intent, employing diverse retrieval strategies, and effectively filtering unintended or inappropriate responses, limiting their effectiveness. This paper introduces Contextual Understanding and Enhanced Search with MLLM (CUE-M), a novel multimodal search framework that addresses these challenges through a multi-stage pipeline comprising image context enrichment, intent refinement, contextual query generation, external API integration, and relevance-based filtering. CUE-M incorporates a robust filtering pipeline combining image-based, text-based, and multimodal classifiers, dynamically adapting to instance- and category-specific concern defined by organizational policies. Extensive experiments on real-word datasets and public benchmarks on knowledge-based VQA and safety demonstrated that CUE-M outperforms baselines and establishes new state-of-the-art results, advancing the capabilities of multimodal retrieval systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_12287 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model Go, Dongyoung Whang, Taesun Lee, Chanhee Kim, Hwa-Yeon Park, Sunghoon Ji, Seunghwan Kim, Jinho Kim, Dongchan Kim, Young-Bum Computation and Language The integration of Retrieval-Augmented Generation (RAG) with Multimodal Large Language Models (MLLMs) has revolutionized information retrieval and expanded the practical applications of AI. However, current systems struggle in accurately interpreting user intent, employing diverse retrieval strategies, and effectively filtering unintended or inappropriate responses, limiting their effectiveness. This paper introduces Contextual Understanding and Enhanced Search with MLLM (CUE-M), a novel multimodal search framework that addresses these challenges through a multi-stage pipeline comprising image context enrichment, intent refinement, contextual query generation, external API integration, and relevance-based filtering. CUE-M incorporates a robust filtering pipeline combining image-based, text-based, and multimodal classifiers, dynamically adapting to instance- and category-specific concern defined by organizational policies. Extensive experiments on real-word datasets and public benchmarks on knowledge-based VQA and safety demonstrated that CUE-M outperforms baselines and establishes new state-of-the-art results, advancing the capabilities of multimodal retrieval systems. |
| title | CUE-M: Contextual Understanding and Enhanced Search with Multimodal Large Language Model |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.12287 |