WGRAMMAR: Leverage Prior Knowledge to Accelerate Structured Decoding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911070673698816 |
|---|---|
| author | Wang, Ran Liu, Xiaoxuan Ren, Hao Chen, Gang Qi, Fanchao Sun, Maosong |
| author_facet | Wang, Ran Liu, Xiaoxuan Ren, Hao Chen, Gang Qi, Fanchao Sun, Maosong |
| contents | Structured decoding enables large language models (LLMs) to generate outputs in formats required by downstream systems, such as HTML or JSON. However, existing methods suffer from efficiency bottlenecks due to grammar compilation, state tracking, and mask creation. We observe that many real-world tasks embed strong prior knowledge about output structure. Leveraging this, we propose a decomposition of constraints into static and dynamic components -- precompiling static structures offline and instantiating dynamic arguments at runtime using grammar snippets. Instead of relying on pushdown automata, we employ a compositional set of operators to model regular formats, achieving lower transition latency. We introduce wgrammar, a lightweight decoding engine that integrates domain-aware simplification, constraint decomposition, and mask caching, achieving up to 250x speedup over existing systems. wgrammar's source code is publicly available at https://github.com/wrran/wgrammar. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_16768 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | WGRAMMAR: Leverage Prior Knowledge to Accelerate Structured Decoding Wang, Ran Liu, Xiaoxuan Ren, Hao Chen, Gang Qi, Fanchao Sun, Maosong Artificial Intelligence Structured decoding enables large language models (LLMs) to generate outputs in formats required by downstream systems, such as HTML or JSON. However, existing methods suffer from efficiency bottlenecks due to grammar compilation, state tracking, and mask creation. We observe that many real-world tasks embed strong prior knowledge about output structure. Leveraging this, we propose a decomposition of constraints into static and dynamic components -- precompiling static structures offline and instantiating dynamic arguments at runtime using grammar snippets. Instead of relying on pushdown automata, we employ a compositional set of operators to model regular formats, achieving lower transition latency. We introduce wgrammar, a lightweight decoding engine that integrates domain-aware simplification, constraint decomposition, and mask caching, achieving up to 250x speedup over existing systems. wgrammar's source code is publicly available at https://github.com/wrran/wgrammar. |
| title | WGRAMMAR: Leverage Prior Knowledge to Accelerate Structured Decoding |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2507.16768 |