Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913547026432000 |
|---|---|
| author | Yao, Jihan Ding, Wenxuan Feng, Shangbin Wang, Lucy Lu Tsvetkov, Yulia |
| author_facet | Yao, Jihan Ding, Wenxuan Feng, Shangbin Wang, Lucy Lu Tsvetkov, Yulia |
| contents | In the absence of abundant reliable annotations for challenging tasks and contexts, how can we expand the frontier of LLM capabilities with potentially wrong answers? We focus on two research questions: (1) Can LLMs generate reliable preferences among wrong options? And if so, (2) Would alignment with such wrong-over-wrong preferences be helpful? We employ methods based on self-consistency, token probabilities, and LLM-as-a-judge to elicit wrong-over-wrong preferences, and fine-tune language models with preference optimization approaches using these synthesized preferences. Extensive experiments with seven LLMs and eight datasets demonstrate that (1) LLMs do have preliminary capability in distinguishing various shades of wrong, achieving up to 20.9% higher performance than random guess; (2) Alignment with wrong-over-wrong preferences helps LLMs to produce less wrong and sometimes even outright correct answers, while overall improving model calibration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11055 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only Yao, Jihan Ding, Wenxuan Feng, Shangbin Wang, Lucy Lu Tsvetkov, Yulia Computation and Language Artificial Intelligence In the absence of abundant reliable annotations for challenging tasks and contexts, how can we expand the frontier of LLM capabilities with potentially wrong answers? We focus on two research questions: (1) Can LLMs generate reliable preferences among wrong options? And if so, (2) Would alignment with such wrong-over-wrong preferences be helpful? We employ methods based on self-consistency, token probabilities, and LLM-as-a-judge to elicit wrong-over-wrong preferences, and fine-tune language models with preference optimization approaches using these synthesized preferences. Extensive experiments with seven LLMs and eight datasets demonstrate that (1) LLMs do have preliminary capability in distinguishing various shades of wrong, achieving up to 20.9% higher performance than random guess; (2) Alignment with wrong-over-wrong preferences helps LLMs to produce less wrong and sometimes even outright correct answers, while overall improving model calibration. |
| title | Varying Shades of Wrong: Aligning LLMs with Wrong Answers Only |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.11055 |