Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.03452 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911345820041216 |
|---|---|
| author | Orbach, Matan Eytan, Ohad Sznajder, Benjamin Gera, Ariel Boni, Odellia Kantor, Yoav Bloch, Gal Levy, Omri Abraham, Hadas Barzilay, Nitzan Shnarch, Eyal Factor, Michael E. Ofek-Koifman, Shila Ta-Shma, Paula Toledo, Assaf |
| author_facet | Orbach, Matan Eytan, Ohad Sznajder, Benjamin Gera, Ariel Boni, Odellia Kantor, Yoav Bloch, Gal Levy, Omri Abraham, Hadas Barzilay, Nitzan Shnarch, Eyal Factor, Michael E. Ofek-Koifman, Shila Ta-Shma, Paula Toledo, Assaf |
| contents | Optimizing Retrieval-Augmented Generation (RAG) configurations for specific tasks is a complex and resource-intensive challenge. Motivated by this challenge, frameworks for RAG hyper-parameter optimization (HPO) have recently emerged, yet their effectiveness has not been rigorously benchmarked. To fill this gap, we present a comprehensive study involving five HPO algorithms over five datasets from diverse domains, including a newly curated real-world product documentation dataset. Our study explores the largest RAG HPO search space to date that includes full grid-search evaluations, and uses three evaluation metrics as optimization targets. Analysis of the results shows that RAG HPO can be done efficiently, either greedily or with random search, and that it significantly boosts RAG performance for all datasets. For greedy HPO approaches, we show that optimizing model selection first is preferable to the common practice of following the RAG pipeline order during optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_03452 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generation Orbach, Matan Eytan, Ohad Sznajder, Benjamin Gera, Ariel Boni, Odellia Kantor, Yoav Bloch, Gal Levy, Omri Abraham, Hadas Barzilay, Nitzan Shnarch, Eyal Factor, Michael E. Ofek-Koifman, Shila Ta-Shma, Paula Toledo, Assaf Computation and Language Artificial Intelligence Machine Learning Optimizing Retrieval-Augmented Generation (RAG) configurations for specific tasks is a complex and resource-intensive challenge. Motivated by this challenge, frameworks for RAG hyper-parameter optimization (HPO) have recently emerged, yet their effectiveness has not been rigorously benchmarked. To fill this gap, we present a comprehensive study involving five HPO algorithms over five datasets from diverse domains, including a newly curated real-world product documentation dataset. Our study explores the largest RAG HPO search space to date that includes full grid-search evaluations, and uses three evaluation metrics as optimization targets. Analysis of the results shows that RAG HPO can be done efficiently, either greedily or with random search, and that it significantly boosts RAG performance for all datasets. For greedy HPO approaches, we show that optimizing model selection first is preferable to the common practice of following the RAG pipeline order during optimization. |
| title | An Analysis of Hyper-Parameter Optimization Methods for Retrieval Augmented Generation |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2505.03452 |