FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909979789754368 |
|---|---|
| author | Xing, Shanli Zhai, Yiyan Jiang, Alexander Dong, Yixin Wu, Yong Ye, Zihao Ruan, Charlie Huang, Yingyi Zhang, Yineng Yin, Liangsheng Bayyapu, Aksara Ceze, Luis Chen, Tianqi |
| author_facet | Xing, Shanli Zhai, Yiyan Jiang, Alexander Dong, Yixin Wu, Yong Ye, Zihao Ruan, Charlie Huang, Yingyi Zhang, Yineng Yin, Liangsheng Bayyapu, Aksara Ceze, Luis Chen, Tianqi |
| contents | Recent advances show that large language models (LLMs) can act as autonomous agents capable of generating GPU kernels, but integrating these AI-generated kernels into real-world inference systems remains challenging. FlashInfer-Bench addresses this gap by establishing a standardized, closed-loop framework that connects kernel generation, benchmarking, and deployment. At its core, FlashInfer Trace provides a unified schema describing kernel definitions, workloads, implementations, and evaluations, enabling consistent communication between agents and systems. Built on real serving traces, FlashInfer-Bench includes a curated dataset, a robust correctness- and performance-aware benchmarking framework, a public leaderboard to track LLM agents' GPU programming capabilities, and a dynamic substitution mechanism (apply()) that seamlessly injects the best-performing kernels into production LLM engines such as SGLang and vLLM. Using FlashInfer-Bench, we further evaluate the performance and limitations of LLM agents, compare the trade-offs among different GPU programming languages, and provide insights for future agent design. FlashInfer-Bench thus establishes a practical, reproducible pathway for continuously improving AI-generated kernels and deploying them into large-scale LLM inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_00227 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems Xing, Shanli Zhai, Yiyan Jiang, Alexander Dong, Yixin Wu, Yong Ye, Zihao Ruan, Charlie Huang, Yingyi Zhang, Yineng Yin, Liangsheng Bayyapu, Aksara Ceze, Luis Chen, Tianqi Artificial Intelligence Recent advances show that large language models (LLMs) can act as autonomous agents capable of generating GPU kernels, but integrating these AI-generated kernels into real-world inference systems remains challenging. FlashInfer-Bench addresses this gap by establishing a standardized, closed-loop framework that connects kernel generation, benchmarking, and deployment. At its core, FlashInfer Trace provides a unified schema describing kernel definitions, workloads, implementations, and evaluations, enabling consistent communication between agents and systems. Built on real serving traces, FlashInfer-Bench includes a curated dataset, a robust correctness- and performance-aware benchmarking framework, a public leaderboard to track LLM agents' GPU programming capabilities, and a dynamic substitution mechanism (apply()) that seamlessly injects the best-performing kernels into production LLM engines such as SGLang and vLLM. Using FlashInfer-Bench, we further evaluate the performance and limitations of LLM agents, compare the trade-offs among different GPU programming languages, and provide insights for future agent design. FlashInfer-Bench thus establishes a practical, reproducible pathway for continuously improving AI-generated kernels and deploying them into large-scale LLM inference. |
| title | FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2601.00227 |