Scaling On-Device GPU Inference for Large Generative Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915268206264320 |
|---|---|
| author | Tang, Jiuqiang Sarokin, Raman Ignasheva, Ekaterina Jensen, Grant Chen, Lin Lee, Juhyun Kulik, Andrei Grundmann, Matthias |
| author_facet | Tang, Jiuqiang Sarokin, Raman Ignasheva, Ekaterina Jensen, Grant Chen, Lin Lee, Juhyun Kulik, Andrei Grundmann, Matthias |
| contents | Driven by the advancements in generative AI, large machine learning models have revolutionized domains such as image processing, audio synthesis, and speech recognition. While server-based deployments remain the locus of peak performance, the imperative for on-device inference, necessitated by privacy and efficiency considerations, persists. Recognizing GPUs as the on-device ML accelerator with the widest reach, we present ML Drift--an optimized framework that extends the capabilities of state-of-the-art GPU-accelerated inference engines. ML Drift enables on-device execution of generative AI workloads which contain 10 to 100x more parameters than existing on-device generative AI models. ML Drift addresses intricate engineering challenges associated with cross-GPU API development, and ensures broad compatibility across mobile and desktop/laptop platforms, thereby facilitating the deployment of significantly more complex models on resource-constrained devices. Our GPU-accelerated ML/AI inference engine achieves an order-of-magnitude performance improvement relative to existing open-source GPU inference engines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_00232 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling On-Device GPU Inference for Large Generative Models Tang, Jiuqiang Sarokin, Raman Ignasheva, Ekaterina Jensen, Grant Chen, Lin Lee, Juhyun Kulik, Andrei Grundmann, Matthias Machine Learning Artificial Intelligence Driven by the advancements in generative AI, large machine learning models have revolutionized domains such as image processing, audio synthesis, and speech recognition. While server-based deployments remain the locus of peak performance, the imperative for on-device inference, necessitated by privacy and efficiency considerations, persists. Recognizing GPUs as the on-device ML accelerator with the widest reach, we present ML Drift--an optimized framework that extends the capabilities of state-of-the-art GPU-accelerated inference engines. ML Drift enables on-device execution of generative AI workloads which contain 10 to 100x more parameters than existing on-device generative AI models. ML Drift addresses intricate engineering challenges associated with cross-GPU API development, and ensures broad compatibility across mobile and desktop/laptop platforms, thereby facilitating the deployment of significantly more complex models on resource-constrained devices. Our GPU-accelerated ML/AI inference engine achieves an order-of-magnitude performance improvement relative to existing open-source GPU inference engines. |
| title | Scaling On-Device GPU Inference for Large Generative Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.00232 |