Self-Consistency Boosts Calibration for Math Reasoning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913265942003712 |
|---|---|
| author | Wang, Ante Song, Linfeng Tian, Ye Peng, Baolin Jin, Lifeng Mi, Haitao Su, Jinsong Yu, Dong |
| author_facet | Wang, Ante Song, Linfeng Tian, Ye Peng, Baolin Jin, Lifeng Mi, Haitao Su, Jinsong Yu, Dong |
| contents | Calibration, which establishes the correlation between accuracy and model confidence, is important for LLM development. We design three off-the-shelf calibration methods based on self-consistency (Wang et al., 2022) for math reasoning tasks. Evaluation on two popular benchmarks (GSM8K and MathQA) using strong open-source LLMs (Mistral and LLaMA2), our methods better bridge model confidence and accuracy than existing methods based on p(True) (Kadavath et al., 2022) or logit (Kadavath et al., 2022). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_09849 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Self-Consistency Boosts Calibration for Math Reasoning Wang, Ante Song, Linfeng Tian, Ye Peng, Baolin Jin, Lifeng Mi, Haitao Su, Jinsong Yu, Dong Computation and Language Artificial Intelligence Calibration, which establishes the correlation between accuracy and model confidence, is important for LLM development. We design three off-the-shelf calibration methods based on self-consistency (Wang et al., 2022) for math reasoning tasks. Evaluation on two popular benchmarks (GSM8K and MathQA) using strong open-source LLMs (Mistral and LLaMA2), our methods better bridge model confidence and accuracy than existing methods based on p(True) (Kadavath et al., 2022) or logit (Kadavath et al., 2022). |
| title | Self-Consistency Boosts Calibration for Math Reasoning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.09849 |