Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.23002 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916973291503616 |
|---|---|
| author | Pang, Lingyou Huang, Lei Lin, Jianyu Wang, Tianyu Horiguchi, Akira Aue, Alexander Priebe, Carey E. |
| author_facet | Pang, Lingyou Huang, Lei Lin, Jianyu Wang, Tianyu Horiguchi, Akira Aue, Alexander Priebe, Carey E. |
| contents | Deploying black-box LLMs requires managing uncertainty in the absence of token-level probability or true labels. We propose introducing an unsupervised conformal inference framework for generation, which integrates: generative models, incorporating: (i) an LLM-compatible atypical score derived from response-embedding Gram matrix, (ii) UCP combined with a bootstrapping variant (BB-UCP) that aggregates residuals to refine quantile precision while maintaining distribution-free, finite-sample coverage, and (iii) conformal alignment, which calibrates a single strictness parameter $τ$ so a user predicate (e.g., factuality lift) holds on unseen batches with probability $\ge 1-α$. Across different benchmark datasets, our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP, while consistently reducing the severity of hallucination, outperforming lightweight per-response detectors with similar computational demands. The result is a label-free, API-compatible gate for test-time filtering that turns geometric signals into calibrated, goal-aligned decisions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23002 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty Pang, Lingyou Huang, Lei Lin, Jianyu Wang, Tianyu Horiguchi, Akira Aue, Alexander Priebe, Carey E. Machine Learning Deploying black-box LLMs requires managing uncertainty in the absence of token-level probability or true labels. We propose introducing an unsupervised conformal inference framework for generation, which integrates: generative models, incorporating: (i) an LLM-compatible atypical score derived from response-embedding Gram matrix, (ii) UCP combined with a bootstrapping variant (BB-UCP) that aggregates residuals to refine quantile precision while maintaining distribution-free, finite-sample coverage, and (iii) conformal alignment, which calibrates a single strictness parameter $τ$ so a user predicate (e.g., factuality lift) holds on unseen batches with probability $\ge 1-α$. Across different benchmark datasets, our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP, while consistently reducing the severity of hallucination, outperforming lightweight per-response detectors with similar computational demands. The result is a label-free, API-compatible gate for test-time filtering that turns geometric signals into calibrated, goal-aligned decisions. |
| title | Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.23002 |