Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.23674 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908529559863296 |
|---|---|
| author | Cheema, Muhammad Taha Aamir, Abeer Muhammad, Khawaja Gul Bhatti, Naveed Anwar Qazi, Ihsan Ayyub Qazi, Zafar Ayyub |
| author_facet | Cheema, Muhammad Taha Aamir, Abeer Muhammad, Khawaja Gul Bhatti, Naveed Anwar Qazi, Ihsan Ayyub Qazi, Zafar Ayyub |
| contents | Large Language Models (LLMs) process millions of queries daily, making efficient response caching a compelling optimization for reducing cost and latency. However, preserving relevance to user queries using this approach proves difficult due to the personalized nature of chatbot interactions and the limited accuracy of semantic similarity search. To address this, we present TweakLLM, a novel routing architecture that employs a lightweight LLM to dynamically adapt cached responses to incoming prompts. Through comprehensive evaluation, including user studies with side-by-side comparisons, satisfaction voting, as well as multi-agent LLM debates, we demonstrate that TweakLLM maintains response quality comparable to frontier models while significantly improving cache effectiveness. Our results across real-world datasets highlight TweakLLM as a scalable, resource-efficient caching solution for high-volume LLM deployments without compromising user experience. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_23674 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses Cheema, Muhammad Taha Aamir, Abeer Muhammad, Khawaja Gul Bhatti, Naveed Anwar Qazi, Ihsan Ayyub Qazi, Zafar Ayyub Machine Learning Computation and Language Large Language Models (LLMs) process millions of queries daily, making efficient response caching a compelling optimization for reducing cost and latency. However, preserving relevance to user queries using this approach proves difficult due to the personalized nature of chatbot interactions and the limited accuracy of semantic similarity search. To address this, we present TweakLLM, a novel routing architecture that employs a lightweight LLM to dynamically adapt cached responses to incoming prompts. Through comprehensive evaluation, including user studies with side-by-side comparisons, satisfaction voting, as well as multi-agent LLM debates, we demonstrate that TweakLLM maintains response quality comparable to frontier models while significantly improving cache effectiveness. Our results across real-world datasets highlight TweakLLM as a scalable, resource-efficient caching solution for high-volume LLM deployments without compromising user experience. |
| title | TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2507.23674 |