Optimizing LLM Queries in Relational Data Analytics Workloads
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912317849993216 |
|---|---|
| author | Liu, Shu Biswal, Asim Kamsetty, Amog Cheng, Audrey Schroeder, Luis Gaspar Patel, Liana Cao, Shiyi Mo, Xiangxi Stoica, Ion Gonzalez, Joseph E. Zaharia, Matei |
| author_facet | Liu, Shu Biswal, Asim Kamsetty, Amog Cheng, Audrey Schroeder, Luis Gaspar Patel, Liana Cao, Shiyi Mo, Xiangxi Stoica, Ion Gonzalez, Joseph E. Zaharia, Matei |
| contents | Batch data analytics is a growing application for Large Language Models (LLMs). LLMs enable users to perform a wide range of natural language tasks, such as classification, entity extraction, and translation, over large datasets. However, LLM inference is highly costly and slow: for example, an NVIDIA L4 GPU running Llama3-8B can only process 6 KB of text per second, taking about a day to handle 15 GB of data; processing a similar amount of data costs around $10K on OpenAI's GPT-4o. In this paper, we propose novel techniques that can significantly reduce the cost of LLM calls for relational data analytics workloads. Our key contribution is developing efficient algorithms for reordering the rows and the fields within each row of an input table to maximize key-value (KV) cache reuse when performing LLM serving. As such, our approach can be easily applied to existing analytics systems and serving platforms. Our evaluation shows that our solution can yield up to 3.4x improvement in job completion time on a benchmark of diverse LLM-based queries using Llama 3 models. Our solution also achieves a 32% cost savings under OpenAI and Anthropic pricing models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_05821 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Optimizing LLM Queries in Relational Data Analytics Workloads Liu, Shu Biswal, Asim Kamsetty, Amog Cheng, Audrey Schroeder, Luis Gaspar Patel, Liana Cao, Shiyi Mo, Xiangxi Stoica, Ion Gonzalez, Joseph E. Zaharia, Matei Machine Learning Databases Batch data analytics is a growing application for Large Language Models (LLMs). LLMs enable users to perform a wide range of natural language tasks, such as classification, entity extraction, and translation, over large datasets. However, LLM inference is highly costly and slow: for example, an NVIDIA L4 GPU running Llama3-8B can only process 6 KB of text per second, taking about a day to handle 15 GB of data; processing a similar amount of data costs around $10K on OpenAI's GPT-4o. In this paper, we propose novel techniques that can significantly reduce the cost of LLM calls for relational data analytics workloads. Our key contribution is developing efficient algorithms for reordering the rows and the fields within each row of an input table to maximize key-value (KV) cache reuse when performing LLM serving. As such, our approach can be easily applied to existing analytics systems and serving platforms. Our evaluation shows that our solution can yield up to 3.4x improvement in job completion time on a benchmark of diverse LLM-based queries using Llama 3 models. Our solution also achieves a 32% cost savings under OpenAI and Anthropic pricing models. |
| title | Optimizing LLM Queries in Relational Data Analytics Workloads |
| topic | Machine Learning Databases |
| url | https://arxiv.org/abs/2403.05821 |