Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2509.20461 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912603507261440 |
|---|---|
| author | Kuwahara, Bruce Lin, Chen-Yuan Huang, Xiao Shi Leung, Kin Kwan Yapeter, Jullian Arta Stanevich, Ilya Perez, Felipe Cresswell, Jesse C. |
| author_facet | Kuwahara, Bruce Lin, Chen-Yuan Huang, Xiao Shi Leung, Kin Kwan Yapeter, Jullian Arta Stanevich, Ilya Perez, Felipe Cresswell, Jesse C. |
| contents | Automatic summarization systems have advanced rapidly with large language models (LLMs), yet they still lack reliable guarantees on inclusion of critical content in high-stakes domains like healthcare, law, and finance. In this work, we introduce Conformal Importance Summarization, the first framework for importance-preserving summary generation which uses conformal prediction to provide rigorous, distribution-free coverage guarantees. By calibrating thresholds on sentence-level importance scores, we enable extractive document summarization with user-specified coverage and recall rates over critical content. Our method is model-agnostic, requires only a small calibration set, and seamlessly integrates with existing black-box LLMs. Experiments on established summarization benchmarks demonstrate that Conformal Importance Summarization achieves the theoretically assured information coverage rate. Our work suggests that Conformal Importance Summarization can be combined with existing techniques to achieve reliable, controllable automatic summarization, paving the way for safer deployment of AI summarization tools in critical applications. Code is available at https://github.com/layer6ai-labs/conformal-importance-summarization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_20461 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Document Summarization with Conformal Importance Guarantees Kuwahara, Bruce Lin, Chen-Yuan Huang, Xiao Shi Leung, Kin Kwan Yapeter, Jullian Arta Stanevich, Ilya Perez, Felipe Cresswell, Jesse C. Computation and Language Machine Learning Automatic summarization systems have advanced rapidly with large language models (LLMs), yet they still lack reliable guarantees on inclusion of critical content in high-stakes domains like healthcare, law, and finance. In this work, we introduce Conformal Importance Summarization, the first framework for importance-preserving summary generation which uses conformal prediction to provide rigorous, distribution-free coverage guarantees. By calibrating thresholds on sentence-level importance scores, we enable extractive document summarization with user-specified coverage and recall rates over critical content. Our method is model-agnostic, requires only a small calibration set, and seamlessly integrates with existing black-box LLMs. Experiments on established summarization benchmarks demonstrate that Conformal Importance Summarization achieves the theoretically assured information coverage rate. Our work suggests that Conformal Importance Summarization can be combined with existing techniques to achieve reliable, controllable automatic summarization, paving the way for safer deployment of AI summarization tools in critical applications. Code is available at https://github.com/layer6ai-labs/conformal-importance-summarization. |
| title | Document Summarization with Conformal Importance Guarantees |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2509.20461 |