TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913745462099968 |
|---|---|
| author | Wu, Feiyang Bian, Zhuohang Duan, Guoyang Xu, Tianle Wu, Junchi Ma, Teng Yao, Yongqiang Gong, Ruihao Zhuo, Youwei |
| author_facet | Wu, Feiyang Bian, Zhuohang Duan, Guoyang Xu, Tianle Wu, Junchi Ma, Teng Yao, Yongqiang Gong, Ruihao Zhuo, Youwei |
| contents | The increasing demand for large language model (LLM) serving has necessitated significant advancements in the optimization and profiling of LLM inference systems. As these models become integral to a wide range of applications, the need for efficient and scalable serving solutions has grown exponentially. This work introduces TokenSim, a comprehensive hardware and software exploration system designed specifically for LLM inference. TokenSim is characterized by its support for extensible system optimizations including scheduling and memory management. We validate the results with systems running with realworld datasets, achieving an error rate of less than 1%. Furthermore, TokenSim facilitates various insightful explorations into the performance and optimization of LLM serving systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_08415 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems Wu, Feiyang Bian, Zhuohang Duan, Guoyang Xu, Tianle Wu, Junchi Ma, Teng Yao, Yongqiang Gong, Ruihao Zhuo, Youwei Distributed, Parallel, and Cluster Computing The increasing demand for large language model (LLM) serving has necessitated significant advancements in the optimization and profiling of LLM inference systems. As these models become integral to a wide range of applications, the need for efficient and scalable serving solutions has grown exponentially. This work introduces TokenSim, a comprehensive hardware and software exploration system designed specifically for LLM inference. TokenSim is characterized by its support for extensible system optimizations including scheduling and memory management. We validate the results with systems running with realworld datasets, achieving an error rate of less than 1%. Furthermore, TokenSim facilitates various insightful explorations into the performance and optimization of LLM serving systems. |
| title | TokenSim: Enabling Hardware and Software Exploration for Large Language Model Inference Systems |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2503.08415 |