Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.18635 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915276974456832 |
|---|---|
| author | Barker, Matthew Bell, Andrew Thomas, Evan Carr, James Andrews, Thomas Bhatt, Umang |
| author_facet | Barker, Matthew Bell, Andrew Thomas, Evan Carr, James Andrews, Thomas Bhatt, Umang |
| contents | While Retrieval Augmented Generation (RAG) has emerged as a popular technique for improving Large Language Model (LLM) systems, it introduces a large number of choices, parameters and hyperparameters that must be made or tuned. This includes the LLM, embedding, and ranker models themselves, as well as hyperparameters governing individual RAG components. Yet, collectively optimizing the entire configuration in a RAG or LLM system remains under-explored - especially in multi-objective settings - due to intractably large solution spaces, noisy objective evaluations, and the high cost of evaluations. In this work, we introduce the first approach for multi-objective parameter optimization of cost, latency, safety and alignment over entire LLM and RAG systems. We find that Bayesian optimization methods significantly outperform baseline approaches, obtaining a superior Pareto front on two new RAG benchmark tasks. We conclude our work with important considerations for practitioners who are designing multi-objective RAG systems, highlighting nuances such as how optimal configurations may not generalize across tasks and objectives. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18635 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems Barker, Matthew Bell, Andrew Thomas, Evan Carr, James Andrews, Thomas Bhatt, Umang Machine Learning Artificial Intelligence Computation and Language 68T20, 68Q32, 90C29, 62P30 I.2.6; I.2.7; G.1.6; G.3 While Retrieval Augmented Generation (RAG) has emerged as a popular technique for improving Large Language Model (LLM) systems, it introduces a large number of choices, parameters and hyperparameters that must be made or tuned. This includes the LLM, embedding, and ranker models themselves, as well as hyperparameters governing individual RAG components. Yet, collectively optimizing the entire configuration in a RAG or LLM system remains under-explored - especially in multi-objective settings - due to intractably large solution spaces, noisy objective evaluations, and the high cost of evaluations. In this work, we introduce the first approach for multi-objective parameter optimization of cost, latency, safety and alignment over entire LLM and RAG systems. We find that Bayesian optimization methods significantly outperform baseline approaches, obtaining a superior Pareto front on two new RAG benchmark tasks. We conclude our work with important considerations for practitioners who are designing multi-objective RAG systems, highlighting nuances such as how optimal configurations may not generalize across tasks and objectives. |
| title | Faster, Cheaper, Better: Multi-Objective Hyperparameter Optimization for LLM and RAG Systems |
| topic | Machine Learning Artificial Intelligence Computation and Language 68T20, 68Q32, 90C29, 62P30 I.2.6; I.2.7; G.1.6; G.3 |
| url | https://arxiv.org/abs/2502.18635 |