Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.08877 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913938849923072 |
|---|---|
| author | Zhang, Hanlong Yang, Jingsheng Li, Hao He, Yuhao Gong, Franck |
| author_facet | Zhang, Hanlong Yang, Jingsheng Li, Hao He, Yuhao Gong, Franck |
| contents | Function Calling is a crucial technique that enables Large Language Models (LLMs) to interact with external systems through APIs. However, the high latency associated with LLM-based Function Calling significantly impacts user experience. This paper presents a novel approach called Oriented Distillation for Inline Acceleration (ODIA) that leverages online user interaction data to accelerate Function Calling. By automatically identifying "simple queries" from production traffic and distilling knowledge from larger models to smaller ones, our method reduces response latency by 45% (expected) and 78% (median) while maintaining accuracy. We demonstrate the effectiveness of our approach through real-world deployment in a music application, where the smaller model successfully handles 60% of traffic with negligible accuracy loss. Our method requires minimal human intervention and continuously improves through automated data collection and model updating, making it a practical solution for production environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_08877 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ODIA: Oriented Distillation for Inline Acceleration of LLM-based Function Calling Zhang, Hanlong Yang, Jingsheng Li, Hao He, Yuhao Gong, Franck Machine Learning Artificial Intelligence Function Calling is a crucial technique that enables Large Language Models (LLMs) to interact with external systems through APIs. However, the high latency associated with LLM-based Function Calling significantly impacts user experience. This paper presents a novel approach called Oriented Distillation for Inline Acceleration (ODIA) that leverages online user interaction data to accelerate Function Calling. By automatically identifying "simple queries" from production traffic and distilling knowledge from larger models to smaller ones, our method reduces response latency by 45% (expected) and 78% (median) while maintaining accuracy. We demonstrate the effectiveness of our approach through real-world deployment in a music application, where the smaller model successfully handles 60% of traffic with negligible accuracy loss. Our method requires minimal human intervention and continuously improves through automated data collection and model updating, making it a practical solution for production environments. |
| title | ODIA: Oriented Distillation for Inline Acceleration of LLM-based Function Calling |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2507.08877 |