IterGen: Iterative Semantic-aware Structured LLM Generation with Backtracking
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909519111520256 |
|---|---|
| author | Ugare, Shubham Gumaste, Rohan Suresh, Tarun Singh, Gagandeep Misailovic, Sasa |
| author_facet | Ugare, Shubham Gumaste, Rohan Suresh, Tarun Singh, Gagandeep Misailovic, Sasa |
| contents | Large Language Models (LLMs) are widely used for tasks such as natural language and code generation, but their outputs often suffer from issues like hallucination, toxicity, and incorrect results. Current libraries for structured LLM generation rely on left-to-right decoding without support for backtracking, limiting the ability to correct or refine outputs mid-generation.
To address this, we introduce IterGen, a user-friendly library for iterative, grammar-guided LLM generation that enables users to move both forward and backward within the generated output based on grammar symbols. By leveraging a symbol-to-position mapping and maintaining the key-value (KV) cache state, IterGen ensures efficient and structured generation while allowing for corrections during the process. We demonstrate IterGen's effectiveness in two important applications: reducing privacy leakage in LLM outputs and improving the accuracy of LLM-generated SQL and Vega-Lite queries.
Our code and additional resources are available at https://structuredllm.com. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07295 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | IterGen: Iterative Semantic-aware Structured LLM Generation with Backtracking Ugare, Shubham Gumaste, Rohan Suresh, Tarun Singh, Gagandeep Misailovic, Sasa Software Engineering Machine Learning Programming Languages Large Language Models (LLMs) are widely used for tasks such as natural language and code generation, but their outputs often suffer from issues like hallucination, toxicity, and incorrect results. Current libraries for structured LLM generation rely on left-to-right decoding without support for backtracking, limiting the ability to correct or refine outputs mid-generation. To address this, we introduce IterGen, a user-friendly library for iterative, grammar-guided LLM generation that enables users to move both forward and backward within the generated output based on grammar symbols. By leveraging a symbol-to-position mapping and maintaining the key-value (KV) cache state, IterGen ensures efficient and structured generation while allowing for corrections during the process. We demonstrate IterGen's effectiveness in two important applications: reducing privacy leakage in LLM outputs and improving the accuracy of LLM-generated SQL and Vega-Lite queries. Our code and additional resources are available at https://structuredllm.com. |
| title | IterGen: Iterative Semantic-aware Structured LLM Generation with Backtracking |
| topic | Software Engineering Machine Learning Programming Languages |
| url | https://arxiv.org/abs/2410.07295 |