Multi-Head RAG: Solving Multi-Aspect Problems with LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910032046587904 |
|---|---|
| author | Besta, Maciej Kubicek, Ales Gerstenberger, Robert Chrapek, Marcin Niggli, Roman Okanovic, Patrik Zhu, Yi Iff, Patrick Podstawski, Michal Weitzendorf, Lucas Chi, Mingyuan Gajda, Joanna Nyczyk, Piotr Müller, Jürgen Niewiadomski, Hubert Hoefler, Torsten |
| author_facet | Besta, Maciej Kubicek, Ales Gerstenberger, Robert Chrapek, Marcin Niggli, Roman Okanovic, Patrik Zhu, Yi Iff, Patrick Podstawski, Michal Weitzendorf, Lucas Chi, Mingyuan Gajda, Joanna Nyczyk, Piotr Müller, Jürgen Niewiadomski, Hubert Hoefler, Torsten |
| contents | Retrieval-Augmented Generation (RAG) improves Large Language Models (LLMs) by retrieving supporting documents into the prompt, but existing methods do not explicitly target queries that require fetching multiple documents with substantially different content. Such multi-aspect queries are challenging because relevant documents can be far apart in embedding space, making joint retrieval difficult. We introduce Multi-Head RAG (MRAG), which addresses this gap with a simple yet powerful idea: using Transformer multi-head attention activations rather than the standard decoder-layer embedding, as retrieval keys. It leverages the observation that different heads capture different semantic aspects. This yields multi-aspect embeddings for both documents and queries, improving retrieval accuracy on complex queries. We show MRAG's design advantages over 18 RAG baselines, up to 20% higher retrieval success ratios for real-world use cases, and improved downstream LLM generation. MRAG integrates seamlessly with existing RAG frameworks and benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_05085 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-Head RAG: Solving Multi-Aspect Problems with LLMs Besta, Maciej Kubicek, Ales Gerstenberger, Robert Chrapek, Marcin Niggli, Roman Okanovic, Patrik Zhu, Yi Iff, Patrick Podstawski, Michal Weitzendorf, Lucas Chi, Mingyuan Gajda, Joanna Nyczyk, Piotr Müller, Jürgen Niewiadomski, Hubert Hoefler, Torsten Computation and Language Artificial Intelligence Information Retrieval Retrieval-Augmented Generation (RAG) improves Large Language Models (LLMs) by retrieving supporting documents into the prompt, but existing methods do not explicitly target queries that require fetching multiple documents with substantially different content. Such multi-aspect queries are challenging because relevant documents can be far apart in embedding space, making joint retrieval difficult. We introduce Multi-Head RAG (MRAG), which addresses this gap with a simple yet powerful idea: using Transformer multi-head attention activations rather than the standard decoder-layer embedding, as retrieval keys. It leverages the observation that different heads capture different semantic aspects. This yields multi-aspect embeddings for both documents and queries, improving retrieval accuracy on complex queries. We show MRAG's design advantages over 18 RAG baselines, up to 20% higher retrieval success ratios for real-world use cases, and improved downstream LLM generation. MRAG integrates seamlessly with existing RAG frameworks and benchmarks. |
| title | Multi-Head RAG: Solving Multi-Aspect Problems with LLMs |
| topic | Computation and Language Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2406.05085 |