Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.26557 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918413307215872 |
|---|---|
| author | Köster, Joris Liu, Zixuan Khajavi, Siavash Zheng, Zizhan |
| author_facet | Köster, Joris Liu, Zixuan Khajavi, Siavash Zheng, Zizhan |
| contents | Large Language Models (LLMs) deliver strong performance but incur high inference cost in real-world services, especially under workloads with repeated or near-duplicate queries across users and sessions. In this work, we propose MemBoost, a memory-boosted LLM serving framework that enables a lightweight model to reuse previously generated answers and retrieve relevant supporting information for cheap inference, while selectively escalating difficult or uncertain queries to a stronger model. Unlike standard retrieval-augmented generation, which primarily grounds a single response, MemBoost is designed for interactive settings by supporting answer reuse, continual memory growth, and cost-aware routing. Experiments across multiple models under simulated workloads show that MemBoost substantially reduces expensive large-model invocations and overall inference cost, while maintaining high answer quality comparable to the strong model baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_26557 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference Köster, Joris Liu, Zixuan Khajavi, Siavash Zheng, Zizhan Computation and Language Large Language Models (LLMs) deliver strong performance but incur high inference cost in real-world services, especially under workloads with repeated or near-duplicate queries across users and sessions. In this work, we propose MemBoost, a memory-boosted LLM serving framework that enables a lightweight model to reuse previously generated answers and retrieve relevant supporting information for cheap inference, while selectively escalating difficult or uncertain queries to a stronger model. Unlike standard retrieval-augmented generation, which primarily grounds a single response, MemBoost is designed for interactive settings by supporting answer reuse, continual memory growth, and cost-aware routing. Experiments across multiple models under simulated workloads show that MemBoost substantially reduces expensive large-model invocations and overall inference cost, while maintaining high answer quality comparable to the strong model baseline. |
| title | MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inference |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.26557 |