CARROT: A Cost Aware Rate Optimal Router
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909616681517056 |
|---|---|
| author | Somerstep, Seamus Polo, Felipe Maia de Oliveira, Allysson Flavio Melo Mangal, Prattyush Silva, Mírian Bhardwaj, Onkar Yurochkin, Mikhail Maity, Subha |
| author_facet | Somerstep, Seamus Polo, Felipe Maia de Oliveira, Allysson Flavio Melo Mangal, Prattyush Silva, Mírian Bhardwaj, Onkar Yurochkin, Mikhail Maity, Subha |
| contents | With the rapid growth in the number of Large Language Models (LLMs), there has been a recent interest in LLM routing, or directing queries to the cheapest LLM that can deliver a suitable response. We conduct a minimax analysis of the routing problem, providing a lower bound and finding that a simple router that predicts both cost and accuracy for each question can be minimax optimal. Inspired by this, we introduce CARROT, a Cost AwaRe Rate Optimal rouTer that selects a model based on estimates of the models' cost and performance. Alongside CARROT, we also introduce the Smart Price-aware ROUTing (SPROUT) dataset to facilitate routing on a wide spectrum of queries with the latest state-of-the-art LLMs. Using SPROUT and prior benchmarks such as Routerbench and open-LLM-leaderboard-v2 we empirically validate CARROT's performance against several alternative routers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03261 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CARROT: A Cost Aware Rate Optimal Router Somerstep, Seamus Polo, Felipe Maia de Oliveira, Allysson Flavio Melo Mangal, Prattyush Silva, Mírian Bhardwaj, Onkar Yurochkin, Mikhail Maity, Subha Machine Learning Networking and Internet Architecture Statistics Theory With the rapid growth in the number of Large Language Models (LLMs), there has been a recent interest in LLM routing, or directing queries to the cheapest LLM that can deliver a suitable response. We conduct a minimax analysis of the routing problem, providing a lower bound and finding that a simple router that predicts both cost and accuracy for each question can be minimax optimal. Inspired by this, we introduce CARROT, a Cost AwaRe Rate Optimal rouTer that selects a model based on estimates of the models' cost and performance. Alongside CARROT, we also introduce the Smart Price-aware ROUTing (SPROUT) dataset to facilitate routing on a wide spectrum of queries with the latest state-of-the-art LLMs. Using SPROUT and prior benchmarks such as Routerbench and open-LLM-leaderboard-v2 we empirically validate CARROT's performance against several alternative routers. |
| title | CARROT: A Cost Aware Rate Optimal Router |
| topic | Machine Learning Networking and Internet Architecture Statistics Theory |
| url | https://arxiv.org/abs/2502.03261 |