Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2412.15803 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917400573640704 |
|---|---|
| author | Ruan, Charlie F. Qin, Yucheng Parthasarathy, Akaash R. Zhou, Xun Lai, Ruihang Jin, Hongyi Dong, Yixin Hou, Bohan Yu, Meng-Shiun Zhai, Yiyan Agarwal, Sudeep Cao, Hangrui Feng, Siyuan Chen, Tianqi |
| author_facet | Ruan, Charlie F. Qin, Yucheng Parthasarathy, Akaash R. Zhou, Xun Lai, Ruihang Jin, Hongyi Dong, Yixin Hou, Bohan Yu, Meng-Shiun Zhai, Yiyan Agarwal, Sudeep Cao, Hangrui Feng, Siyuan Chen, Tianqi |
| contents | Advancements in large language models (LLMs) have unlocked remarkable capabilities. While deploying these models typically requires server-grade GPUs and cloud-based inference, the recent emergence of smaller open-source models and increasingly powerful consumer devices have made on-device deployment practical. The web browser as a platform for on-device deployment is universally accessible, provides a natural agentic environment, and conveniently abstracts out the different backends from diverse device vendors. To address this opportunity, we introduce WebLLM, an open-source JavaScript framework that enables high-performance LLM inference entirely within web browsers. WebLLM provides an OpenAI-style API for seamless integration into web applications, and leverages WebGPU for efficient local GPU acceleration and WebAssembly for performant CPU computation. With machine learning compilers MLC-LLM and Apache TVM, WebLLM leverages optimized WebGPU kernels, overcoming the absence of performant WebGPU kernel libraries. Evaluations show that WebLLM can retain up to 80% native performance on the same device, with room to further close the gap. WebLLM paves the way for universally accessible, privacy-preserving, personalized, and locally powered LLM applications in web browsers. The code is available at: https://github.com/mlc-ai/web-llm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_15803 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WebLLM: A High-Performance In-Browser LLM Inference Engine Ruan, Charlie F. Qin, Yucheng Parthasarathy, Akaash R. Zhou, Xun Lai, Ruihang Jin, Hongyi Dong, Yixin Hou, Bohan Yu, Meng-Shiun Zhai, Yiyan Agarwal, Sudeep Cao, Hangrui Feng, Siyuan Chen, Tianqi Machine Learning Artificial Intelligence Advancements in large language models (LLMs) have unlocked remarkable capabilities. While deploying these models typically requires server-grade GPUs and cloud-based inference, the recent emergence of smaller open-source models and increasingly powerful consumer devices have made on-device deployment practical. The web browser as a platform for on-device deployment is universally accessible, provides a natural agentic environment, and conveniently abstracts out the different backends from diverse device vendors. To address this opportunity, we introduce WebLLM, an open-source JavaScript framework that enables high-performance LLM inference entirely within web browsers. WebLLM provides an OpenAI-style API for seamless integration into web applications, and leverages WebGPU for efficient local GPU acceleration and WebAssembly for performant CPU computation. With machine learning compilers MLC-LLM and Apache TVM, WebLLM leverages optimized WebGPU kernels, overcoming the absence of performant WebGPU kernel libraries. Evaluations show that WebLLM can retain up to 80% native performance on the same device, with room to further close the gap. WebLLM paves the way for universally accessible, privacy-preserving, personalized, and locally powered LLM applications in web browsers. The code is available at: https://github.com/mlc-ai/web-llm. |
| title | WebLLM: A High-Performance In-Browser LLM Inference Engine |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2412.15803 |