CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Tasks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912473827770368 |
|---|---|
| author | Besta, Maciej Paleari, Lorenzo Copik, Marcin Gerstenberger, Robert Kubicek, Ales Nyczyk, Piotr Iff, Patrick Schreiber, Eric Srindran, Tanja Lehmann, Tomasz Niewiadomski, Hubert Hoefler, Torsten |
| author_facet | Besta, Maciej Paleari, Lorenzo Copik, Marcin Gerstenberger, Robert Kubicek, Ales Nyczyk, Piotr Iff, Patrick Schreiber, Eric Srindran, Tanja Lehmann, Tomasz Niewiadomski, Hubert Hoefler, Torsten |
| contents | Large Language Models (LLMs) are transforming a wide range of domains, yet verifying their outputs remains a significant challenge, especially for complex open-ended tasks such as consolidation, summarization, and knowledge extraction. To address this, we introduce CheckEmbed (CE): a simple, scalable, and accurate verification method. CE reduces each LLM answer to a single embedding vector using powerful modern embedding LLM models like SFR-Embedding-Mistral. Prior methods such as BERTScore and SelfCheckGPT relied on weaker encoders like BERT, forcing them to operate at token or sentence granularity. In contrast, CE performs fast, semantically rich comparisons directly at the whole-answer level, overcoming key limitations in both accuracy and scalability. We conduct a comprehensive design and time complexity analysis across 13 verification baselines, including classical text scorers (e.g., BLEU), stability-based methods (e.g., SelfCheckGPT), and generative evaluators (e.g., LLM-as-a-Judge), which highlights the effectiveness, efficiency, versatility, and simplicity of CE. Empirical results show that CE reliably detects hallucinations in both closed and open-ended tasks. We further present evidence that CE generalizes beyond text to other modalities such as vision, establishing it as a practical and versatile verification framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_02524 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Tasks Besta, Maciej Paleari, Lorenzo Copik, Marcin Gerstenberger, Robert Kubicek, Ales Nyczyk, Piotr Iff, Patrick Schreiber, Eric Srindran, Tanja Lehmann, Tomasz Niewiadomski, Hubert Hoefler, Torsten Computation and Language Large Language Models (LLMs) are transforming a wide range of domains, yet verifying their outputs remains a significant challenge, especially for complex open-ended tasks such as consolidation, summarization, and knowledge extraction. To address this, we introduce CheckEmbed (CE): a simple, scalable, and accurate verification method. CE reduces each LLM answer to a single embedding vector using powerful modern embedding LLM models like SFR-Embedding-Mistral. Prior methods such as BERTScore and SelfCheckGPT relied on weaker encoders like BERT, forcing them to operate at token or sentence granularity. In contrast, CE performs fast, semantically rich comparisons directly at the whole-answer level, overcoming key limitations in both accuracy and scalability. We conduct a comprehensive design and time complexity analysis across 13 verification baselines, including classical text scorers (e.g., BLEU), stability-based methods (e.g., SelfCheckGPT), and generative evaluators (e.g., LLM-as-a-Judge), which highlights the effectiveness, efficiency, versatility, and simplicity of CE. Empirical results show that CE reliably detects hallucinations in both closed and open-ended tasks. We further present evidence that CE generalizes beyond text to other modalities such as vision, establishing it as a practical and versatile verification framework. |
| title | CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Tasks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.02524 |