AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911935177424896 |
|---|---|
| author | Fu, Jia Qin, Xiaoting Yang, Fangkai Wang, Lu Zhang, Jue Lin, Qingwei Chen, Yubo Zhang, Dongmei Rajmohan, Saravan Zhang, Qi |
| author_facet | Fu, Jia Qin, Xiaoting Yang, Fangkai Wang, Lu Zhang, Jue Lin, Qingwei Chen, Yubo Zhang, Dongmei Rajmohan, Saravan Zhang, Qi |
| contents | Recent advancements in Large Language Models have transformed ML/AI development, necessitating a reevaluation of AutoML principles for the Retrieval-Augmented Generation (RAG) systems. To address the challenges of hyper-parameter optimization and online adaptation in RAG, we propose the AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical MAB (Hier-MAB) method for efficient exploration of large search spaces. We conduct extensive experiments on tuning hyper-parameters, such as top-k retrieved documents, prompt compression ratio, and embedding methods, using the ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly optimization all three hyper-parameters demonstrate that MAB-based online learning methods can achieve Recall@5 $\approx 0.8$ for scenarios with prominent gradients in search space, using only $\sim20\%$ of the LLM API calls required by the Grid Search approach. Additionally, the proposed Hier-MAB approach outperforms other baselines in more challenging optimization scenarios. The code will be made available at https://aka.ms/autorag. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19251 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation Fu, Jia Qin, Xiaoting Yang, Fangkai Wang, Lu Zhang, Jue Lin, Qingwei Chen, Yubo Zhang, Dongmei Rajmohan, Saravan Zhang, Qi Computation and Language Artificial Intelligence Recent advancements in Large Language Models have transformed ML/AI development, necessitating a reevaluation of AutoML principles for the Retrieval-Augmented Generation (RAG) systems. To address the challenges of hyper-parameter optimization and online adaptation in RAG, we propose the AutoRAG-HP framework, which formulates the hyper-parameter tuning as an online multi-armed bandit (MAB) problem and introduces a novel two-level Hierarchical MAB (Hier-MAB) method for efficient exploration of large search spaces. We conduct extensive experiments on tuning hyper-parameters, such as top-k retrieved documents, prompt compression ratio, and embedding methods, using the ALCE-ASQA and Natural Questions datasets. Our evaluation from jointly optimization all three hyper-parameters demonstrate that MAB-based online learning methods can achieve Recall@5 $\approx 0.8$ for scenarios with prominent gradients in search space, using only $\sim20\%$ of the LLM API calls required by the Grid Search approach. Additionally, the proposed Hier-MAB approach outperforms other baselines in more challenging optimization scenarios. The code will be made available at https://aka.ms/autorag. |
| title | AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.19251 |