Towards a Probabilistic Framework for Analyzing and Improving LLM-Enabled Software
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910910237376512 |
|---|---|
| author | Baldonado, Juan Manuel Bonomo-Braberman, Flavia Braberman, Víctor Adrián |
| author_facet | Baldonado, Juan Manuel Bonomo-Braberman, Flavia Braberman, Víctor Adrián |
| contents | Ensuring the reliability and verifiability of large language model (LLM)-enabled systems remains a significant challenge in software engineering. We propose a probabilistic framework for systematically analyzing and improving these systems by modeling and refining distributions over clusters of semantically equivalent outputs. This framework facilitates the evaluation and iterative improvement of Transference Models--key software components that utilize LLMs to transform inputs into outputs for downstream tasks. To illustrate its utility, we apply the framework to the autoformalization problem, where natural language documentation is transformed into formal program specifications. Our case illustrates how distribution-aware analysis enables the identification of weaknesses and guides focused alignment improvements, resulting in more reliable and interpretable outputs. This principled approach offers a foundation for addressing critical challenges in the development of robust LLM-enabled systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_06370 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards a Probabilistic Framework for Analyzing and Improving LLM-Enabled Software Baldonado, Juan Manuel Bonomo-Braberman, Flavia Braberman, Víctor Adrián Software Engineering Artificial Intelligence D.2.5; D.2.4; I.2.7 Ensuring the reliability and verifiability of large language model (LLM)-enabled systems remains a significant challenge in software engineering. We propose a probabilistic framework for systematically analyzing and improving these systems by modeling and refining distributions over clusters of semantically equivalent outputs. This framework facilitates the evaluation and iterative improvement of Transference Models--key software components that utilize LLMs to transform inputs into outputs for downstream tasks. To illustrate its utility, we apply the framework to the autoformalization problem, where natural language documentation is transformed into formal program specifications. Our case illustrates how distribution-aware analysis enables the identification of weaknesses and guides focused alignment improvements, resulting in more reliable and interpretable outputs. This principled approach offers a foundation for addressing critical challenges in the development of robust LLM-enabled systems. |
| title | Towards a Probabilistic Framework for Analyzing and Improving LLM-Enabled Software |
| topic | Software Engineering Artificial Intelligence D.2.5; D.2.4; I.2.7 |
| url | https://arxiv.org/abs/2501.06370 |