Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929479078641664 |
|---|---|
| author | Agrawal, Amey Agarwal, Anmol Kedia, Nitin Mohan, Jayashree Kundu, Souvik Kwatra, Nipun Ramjee, Ramachandran Tumanov, Alexey |
| author_facet | Agrawal, Amey Agarwal, Anmol Kedia, Nitin Mohan, Jayashree Kundu, Souvik Kwatra, Nipun Ramjee, Ramachandran Tumanov, Alexey |
| contents | Serving large language models (LLMs) in production can incur substantial costs, which has prompted recent advances in inference system optimizations. Today, these systems are evaluated against conventional latency and throughput metrics (eg. TTFT, TBT, Normalised Latency and TPOT). However, these metrics fail to fully capture the nuances of LLM inference, leading to an incomplete assessment of user-facing performance crucial for real-time applications such as chat and translation. In this paper, we first identify the pitfalls of current performance metrics in evaluating LLM inference systems. We then propose Etalon, a comprehensive performance evaluation framework that includes fluidity-index -- a novel metric designed to reflect the intricacies of the LLM inference process and its impact on real-time user experience. Finally, we evaluate various existing open-source platforms and model-as-a-service offerings using Etalon, discussing their strengths and weaknesses. Etalon is available at https://github.com/project-etalon/etalon. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_07000 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems Agrawal, Amey Agarwal, Anmol Kedia, Nitin Mohan, Jayashree Kundu, Souvik Kwatra, Nipun Ramjee, Ramachandran Tumanov, Alexey Machine Learning Artificial Intelligence Computation and Language Distributed, Parallel, and Cluster Computing Serving large language models (LLMs) in production can incur substantial costs, which has prompted recent advances in inference system optimizations. Today, these systems are evaluated against conventional latency and throughput metrics (eg. TTFT, TBT, Normalised Latency and TPOT). However, these metrics fail to fully capture the nuances of LLM inference, leading to an incomplete assessment of user-facing performance crucial for real-time applications such as chat and translation. In this paper, we first identify the pitfalls of current performance metrics in evaluating LLM inference systems. We then propose Etalon, a comprehensive performance evaluation framework that includes fluidity-index -- a novel metric designed to reflect the intricacies of the LLM inference process and its impact on real-time user experience. Finally, we evaluate various existing open-source platforms and model-as-a-service offerings using Etalon, discussing their strengths and weaknesses. Etalon is available at https://github.com/project-etalon/etalon. |
| title | Etalon: Holistic Performance Evaluation Framework for LLM Inference Systems |
| topic | Machine Learning Artificial Intelligence Computation and Language Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2407.07000 |