Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.06988 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911793360666624 |
|---|---|
| author | Beurer-Kellner, Luca Fischer, Marc Vechev, Martin |
| author_facet | Beurer-Kellner, Luca Fischer, Marc Vechev, Martin |
| contents | To ensure that text generated by large language models (LLMs) is in an expected format, constrained decoding proposes to enforce strict formal language constraints during generation. However, as we show in this work, not only do such methods incur performance overhead during generation, but many of them also significantly impair task accuracy, if they do not correctly align the underlying LLM sub-word vocabularies with external constraints. To address this, we present a novel decoding algorithm, DOMINO, that can enforce constraints in a fully subword-aligned fashion, while leveraging pre-computation and speculative decoding to achieve virtually no overhead and in some cases even almost 2$\times$ speedup over unconstrained decoding -- thereby outperforming existing approaches by a wide margin. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_06988 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation Beurer-Kellner, Luca Fischer, Marc Vechev, Martin Machine Learning Computation and Language To ensure that text generated by large language models (LLMs) is in an expected format, constrained decoding proposes to enforce strict formal language constraints during generation. However, as we show in this work, not only do such methods incur performance overhead during generation, but many of them also significantly impair task accuracy, if they do not correctly align the underlying LLM sub-word vocabularies with external constraints. To address this, we present a novel decoding algorithm, DOMINO, that can enforce constraints in a fully subword-aligned fashion, while leveraging pre-computation and speculative decoding to achieve virtually no overhead and in some cases even almost 2$\times$ speedup over unconstrained decoding -- thereby outperforming existing approaches by a wide margin. |
| title | Guiding LLMs The Right Way: Fast, Non-Invasive Constrained Generation |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2403.06988 |