CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913402374324224 |
|---|---|
| author | Hsu, I-Hung Wang, Zifeng Le, Long T. Miculicich, Lesly Peng, Nanyun Lee, Chen-Yu Pfister, Tomas |
| author_facet | Hsu, I-Hung Wang, Zifeng Le, Long T. Miculicich, Lesly Peng, Nanyun Lee, Chen-Yu Pfister, Tomas |
| contents | Grounded generation aims to equip language models (LMs) with the ability to produce more credible and accountable responses by accurately citing verifiable sources. However, existing methods, by either feeding LMs with raw or preprocessed materials, remain prone to errors. To address this, we introduce CaLM, a novel verification framework. CaLM leverages the insight that a robust grounded response should be consistent with information derived solely from its cited sources. Our framework empowers smaller LMs, which rely less on parametric memory and excel at processing relevant information given a query, to validate the output of larger LMs. Larger LM responses that closely align with the smaller LMs' output, which relies exclusively on cited documents, are verified. Responses showing discrepancies are iteratively refined through a feedback loop. Experiments on three open-domain question-answering datasets demonstrate significant performance gains of 1.5% to 7% absolute average without any required model fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_05365 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation Hsu, I-Hung Wang, Zifeng Le, Long T. Miculicich, Lesly Peng, Nanyun Lee, Chen-Yu Pfister, Tomas Computation and Language Artificial Intelligence Machine Learning Grounded generation aims to equip language models (LMs) with the ability to produce more credible and accountable responses by accurately citing verifiable sources. However, existing methods, by either feeding LMs with raw or preprocessed materials, remain prone to errors. To address this, we introduce CaLM, a novel verification framework. CaLM leverages the insight that a robust grounded response should be consistent with information derived solely from its cited sources. Our framework empowers smaller LMs, which rely less on parametric memory and excel at processing relevant information given a query, to validate the output of larger LMs. Larger LM responses that closely align with the smaller LMs' output, which relies exclusively on cited documents, are verified. Responses showing discrepancies are iteratively refined through a feedback loop. Experiments on three open-domain question-answering datasets demonstrate significant performance gains of 1.5% to 7% absolute average without any required model fine-tuning. |
| title | CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2406.05365 |