DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929231445884928 |
|---|---|
| author | Singhvi, Arnav Shetty, Manish Tan, Shangyin Potts, Christopher Sen, Koushik Zaharia, Matei Khattab, Omar |
| author_facet | Singhvi, Arnav Shetty, Manish Tan, Shangyin Potts, Christopher Sen, Koushik Zaharia, Matei Khattab, Omar |
| contents | Chaining language model (LM) calls as composable modules is fueling a new way of programming, but ensuring LMs adhere to important constraints requires heuristic "prompt engineering". We introduce LM Assertions, a programming construct for expressing computational constraints that LMs should satisfy. We integrate our constructs into the recent DSPy programming model for LMs, and present new strategies that allow DSPy to compile programs with LM Assertions into more reliable and accurate systems. We also propose strategies to use assertions at inference time for automatic self-refinement with LMs. We report on four diverse case studies for text generation and find that LM Assertions improve not only compliance with imposed rules but also downstream task performance, passing constraints up to 164% more often and generating up to 37% more higher-quality responses. Our reference implementation of LM Assertions is integrated into DSPy at https://github.com/stanfordnlp/dspy |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_13382 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines Singhvi, Arnav Shetty, Manish Tan, Shangyin Potts, Christopher Sen, Koushik Zaharia, Matei Khattab, Omar Computation and Language Artificial Intelligence Programming Languages Chaining language model (LM) calls as composable modules is fueling a new way of programming, but ensuring LMs adhere to important constraints requires heuristic "prompt engineering". We introduce LM Assertions, a programming construct for expressing computational constraints that LMs should satisfy. We integrate our constructs into the recent DSPy programming model for LMs, and present new strategies that allow DSPy to compile programs with LM Assertions into more reliable and accurate systems. We also propose strategies to use assertions at inference time for automatic self-refinement with LMs. We report on four diverse case studies for text generation and find that LM Assertions improve not only compliance with imposed rules but also downstream task performance, passing constraints up to 164% more often and generating up to 37% more higher-quality responses. Our reference implementation of LM Assertions is integrated into DSPy at https://github.com/stanfordnlp/dspy |
| title | DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelines |
| topic | Computation and Language Artificial Intelligence Programming Languages |
| url | https://arxiv.org/abs/2312.13382 |