SGLang: Efficient Execution of Structured Language Model Programs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913379415752704 |
|---|---|
| author | Zheng, Lianmin Yin, Liangsheng Xie, Zhiqiang Sun, Chuyue Huang, Jeff Yu, Cody Hao Cao, Shiyi Kozyrakis, Christos Stoica, Ion Gonzalez, Joseph E. Barrett, Clark Sheng, Ying |
| author_facet | Zheng, Lianmin Yin, Liangsheng Xie, Zhiqiang Sun, Chuyue Huang, Jeff Yu, Cody Hao Cao, Shiyi Kozyrakis, Christos Stoica, Ion Gonzalez, Joseph E. Barrett, Clark Sheng, Ying |
| contents | Large language models (LLMs) are increasingly used for complex tasks that require multiple generation calls, advanced prompting techniques, control flow, and structured inputs/outputs. However, efficient systems are lacking for programming and executing these applications. We introduce SGLang, a system for efficient execution of complex language model programs. SGLang consists of a frontend language and a runtime. The frontend simplifies programming with primitives for generation and parallelism control. The runtime accelerates execution with novel optimizations like RadixAttention for KV cache reuse and compressed finite state machines for faster structured output decoding. Experiments show that SGLang achieves up to 6.4x higher throughput compared to state-of-the-art inference systems on various large language and multi-modal models on tasks including agent control, logical reasoning, few-shot learning benchmarks, JSON decoding, retrieval-augmented generation pipelines, and multi-turn chat. The code is publicly available at https://github.com/sgl-project/sglang |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_07104 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SGLang: Efficient Execution of Structured Language Model Programs Zheng, Lianmin Yin, Liangsheng Xie, Zhiqiang Sun, Chuyue Huang, Jeff Yu, Cody Hao Cao, Shiyi Kozyrakis, Christos Stoica, Ion Gonzalez, Joseph E. Barrett, Clark Sheng, Ying Artificial Intelligence Programming Languages Large language models (LLMs) are increasingly used for complex tasks that require multiple generation calls, advanced prompting techniques, control flow, and structured inputs/outputs. However, efficient systems are lacking for programming and executing these applications. We introduce SGLang, a system for efficient execution of complex language model programs. SGLang consists of a frontend language and a runtime. The frontend simplifies programming with primitives for generation and parallelism control. The runtime accelerates execution with novel optimizations like RadixAttention for KV cache reuse and compressed finite state machines for faster structured output decoding. Experiments show that SGLang achieves up to 6.4x higher throughput compared to state-of-the-art inference systems on various large language and multi-modal models on tasks including agent control, logical reasoning, few-shot learning benchmarks, JSON decoding, retrieval-augmented generation pipelines, and multi-turn chat. The code is publicly available at https://github.com/sgl-project/sglang |
| title | SGLang: Efficient Execution of Structured Language Model Programs |
| topic | Artificial Intelligence Programming Languages |
| url | https://arxiv.org/abs/2312.07104 |