Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2501.02342 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913636376641536 |
|---|---|
| author | Khiabani, Yahya Sowti Atif, Farris Hsu, Chieh Stahlmann, Sven Michels, Tobias Kramer, Sebastian Heidrich, Benedikt Sarfraz, M. Saquib Merten, Julian Tafazzoli, Faezeh |
| author_facet | Khiabani, Yahya Sowti Atif, Farris Hsu, Chieh Stahlmann, Sven Michels, Tobias Kramer, Sebastian Heidrich, Benedikt Sarfraz, M. Saquib Merten, Julian Tafazzoli, Faezeh |
| contents | We propose a holistic approach for deploying Small Language Models (SLMs) as function-calling agents within vehicles as edge devices, offering a more flexible and robust alternative to traditional rule-based systems. By leveraging SLMs, we simplify vehicle control mechanisms and enhance the user experience. Given the in-vehicle hardware constraints, we apply state-of-the-art model compression techniques, including structured pruning, healing, and quantization, ensuring that the model fits within the resource limitations while maintaining acceptable performance. Our work focuses on optimizing a representative SLM, Microsoft's Phi-3 mini, and outlines best practices for enabling embedded models, including compression, task-specific fine-tuning, and vehicle integration. We demonstrate that, despite significant reduction in model size which removes up to 2 billion parameters from the original model, our approach preserves the model's ability to handle complex in-vehicle tasks accurately and efficiently. Furthermore, by executing the model in a lightweight runtime environment, we achieve a generation speed of 11 tokens per second, making real-time, on-device inference feasible without hardware acceleration. Our results demonstrate the potential of SLMs to transform vehicle control systems, enabling more intuitive interactions between users and their vehicles for an enhanced driving experience. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_02342 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Optimizing Small Language Models for In-Vehicle Function-Calling Khiabani, Yahya Sowti Atif, Farris Hsu, Chieh Stahlmann, Sven Michels, Tobias Kramer, Sebastian Heidrich, Benedikt Sarfraz, M. Saquib Merten, Julian Tafazzoli, Faezeh Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction We propose a holistic approach for deploying Small Language Models (SLMs) as function-calling agents within vehicles as edge devices, offering a more flexible and robust alternative to traditional rule-based systems. By leveraging SLMs, we simplify vehicle control mechanisms and enhance the user experience. Given the in-vehicle hardware constraints, we apply state-of-the-art model compression techniques, including structured pruning, healing, and quantization, ensuring that the model fits within the resource limitations while maintaining acceptable performance. Our work focuses on optimizing a representative SLM, Microsoft's Phi-3 mini, and outlines best practices for enabling embedded models, including compression, task-specific fine-tuning, and vehicle integration. We demonstrate that, despite significant reduction in model size which removes up to 2 billion parameters from the original model, our approach preserves the model's ability to handle complex in-vehicle tasks accurately and efficiently. Furthermore, by executing the model in a lightweight runtime environment, we achieve a generation speed of 11 tokens per second, making real-time, on-device inference feasible without hardware acceleration. Our results demonstrate the potential of SLMs to transform vehicle control systems, enabling more intuitive interactions between users and their vehicles for an enhanced driving experience. |
| title | Optimizing Small Language Models for In-Vehicle Function-Calling |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2501.02342 |