TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908445850992640 |
|---|---|
| author | Yaldiz, Duygu Nur Bakman, Yavuz Faruk Kang, Sungmin Öziş, Alperen Yildiz, Hayrettin Eren Shah, Mitash Ashish Huang, Zhiqi Kumar, Anoop Samuel, Alfy Liu, Daben Karimireddy, Sai Praneeth Avestimehr, Salman |
| author_facet | Yaldiz, Duygu Nur Bakman, Yavuz Faruk Kang, Sungmin Öziş, Alperen Yildiz, Hayrettin Eren Shah, Mitash Ashish Huang, Zhiqi Kumar, Anoop Samuel, Alfy Liu, Daben Karimireddy, Sai Praneeth Avestimehr, Salman |
| contents | Generative Large Language Models (LLMs)inevitably produce untruthful responses. Accurately predicting the truthfulness of these outputs is critical, especially in high-stakes settings. To accelerate research in this domain and make truthfulness prediction methods more accessible, we introduce TruthTorchLM an open-source, comprehensive Python library featuring over 30 truthfulness prediction methods, which we refer to as Truth Methods. Unlike existing toolkits such as Guardrails, which focus solely on document-grounded verification, or LM-Polygraph, which is limited to uncertainty-based methods, TruthTorchLM offers a broad and extensible collection of techniques. These methods span diverse tradeoffs in computational cost, access level (e.g., black-box vs white-box), grounding document requirements, and supervision type (self-supervised or supervised). TruthTorchLM is seamlessly compatible with both HuggingFace and LiteLLM, enabling support for locally hosted and API-based models. It also provides a unified interface for generation, evaluation, calibration, and long-form truthfulness prediction, along with a flexible framework for extending the library with new methods. We conduct an evaluation of representative truth methods on three datasets, TriviaQA, GSM8K, and FactScore-Bio. The code is available at https://github.com/Ybakman/TruthTorchLM |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_08203 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs Yaldiz, Duygu Nur Bakman, Yavuz Faruk Kang, Sungmin Öziş, Alperen Yildiz, Hayrettin Eren Shah, Mitash Ashish Huang, Zhiqi Kumar, Anoop Samuel, Alfy Liu, Daben Karimireddy, Sai Praneeth Avestimehr, Salman Computation and Language Generative Large Language Models (LLMs)inevitably produce untruthful responses. Accurately predicting the truthfulness of these outputs is critical, especially in high-stakes settings. To accelerate research in this domain and make truthfulness prediction methods more accessible, we introduce TruthTorchLM an open-source, comprehensive Python library featuring over 30 truthfulness prediction methods, which we refer to as Truth Methods. Unlike existing toolkits such as Guardrails, which focus solely on document-grounded verification, or LM-Polygraph, which is limited to uncertainty-based methods, TruthTorchLM offers a broad and extensible collection of techniques. These methods span diverse tradeoffs in computational cost, access level (e.g., black-box vs white-box), grounding document requirements, and supervision type (self-supervised or supervised). TruthTorchLM is seamlessly compatible with both HuggingFace and LiteLLM, enabling support for locally hosted and API-based models. It also provides a unified interface for generation, evaluation, calibration, and long-form truthfulness prediction, along with a flexible framework for extending the library with new methods. We conduct an evaluation of representative truth methods on three datasets, TriviaQA, GSM8K, and FactScore-Bio. The code is available at https://github.com/Ybakman/TruthTorchLM |
| title | TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.08203 |