Conformal Prediction Sets for Deep Generative Models via Reduction to Conformal Regression
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908410202554368 |
|---|---|
| author | Shahrokhi, Hooman Roy, Devjeet Raj Yan, Yan Arnaoudova, Venera Doppa, Janaradhan Rao |
| author_facet | Shahrokhi, Hooman Roy, Devjeet Raj Yan, Yan Arnaoudova, Venera Doppa, Janaradhan Rao |
| contents | We consider the problem of generating valid and small prediction sets by sampling outputs (e.g., software code and natural language text) from a black-box deep generative model for a given input (e.g., textual prompt). The validity of a prediction set is determined by a user-defined binary admissibility function depending on the target application. For example, requiring at least one program in the set to pass all test cases in code generation application. To address this problem, we develop a simple and effective conformal inference algorithm referred to as Generative Prediction Sets (GPS). Given a set of calibration examples and black-box access to a deep generative model, GPS can generate prediction sets with provable guarantees. The key insight behind GPS is to exploit the inherent structure within the distribution over the minimum number of samples needed to obtain an admissible output to develop a simple conformal regression approach over the minimum number of samples. Experiments on multiple datasets for code and math word problems using different large language models demonstrate the efficacy of GPS over state-of-the-art methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10512 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Conformal Prediction Sets for Deep Generative Models via Reduction to Conformal Regression Shahrokhi, Hooman Roy, Devjeet Raj Yan, Yan Arnaoudova, Venera Doppa, Janaradhan Rao Machine Learning Artificial Intelligence We consider the problem of generating valid and small prediction sets by sampling outputs (e.g., software code and natural language text) from a black-box deep generative model for a given input (e.g., textual prompt). The validity of a prediction set is determined by a user-defined binary admissibility function depending on the target application. For example, requiring at least one program in the set to pass all test cases in code generation application. To address this problem, we develop a simple and effective conformal inference algorithm referred to as Generative Prediction Sets (GPS). Given a set of calibration examples and black-box access to a deep generative model, GPS can generate prediction sets with provable guarantees. The key insight behind GPS is to exploit the inherent structure within the distribution over the minimum number of samples needed to obtain an admissible output to develop a simple conformal regression approach over the minimum number of samples. Experiments on multiple datasets for code and math word problems using different large language models demonstrate the efficacy of GPS over state-of-the-art methods. |
| title | Conformal Prediction Sets for Deep Generative Models via Reduction to Conformal Regression |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2503.10512 |