Predicting Text Preference Via Structured Comparative Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911938914549760 |
|---|---|
| author | Yan, Jing Nathan Liu, Tianqi Chiu, Justin T Shen, Jiaming Qin, Zhen Yu, Yue Zhao, Yao Lakshmanan, Charu Kurzion, Yair Rush, Alexander M. Liu, Jialu Bendersky, Michael |
| author_facet | Yan, Jing Nathan Liu, Tianqi Chiu, Justin T Shen, Jiaming Qin, Zhen Yu, Yue Zhao, Yao Lakshmanan, Charu Kurzion, Yair Rush, Alexander M. Liu, Jialu Bendersky, Michael |
| contents | Comparative reasoning plays a crucial role in text preference prediction; however, large language models (LLMs) often demonstrate inconsistencies in their reasoning. While approaches like Chain-of-Thought improve accuracy in many other settings, they struggle to consistently distinguish the similarities and differences of complex texts. We introduce SC, a prompting approach that predicts text preferences by generating structured intermediate comparisons. SC begins by proposing aspects of comparison, followed by generating textual comparisons under each aspect. We select consistent comparisons with a pairwise consistency comparator that ensures each aspect's comparisons clearly distinguish differences between texts, significantly reducing hallucination and improving consistency. Our comprehensive evaluations across various NLP tasks, including summarization, retrieval, and automatic rating, demonstrate that SC equips LLMs to achieve state-of-the-art performance in text preference prediction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_08390 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Predicting Text Preference Via Structured Comparative Reasoning Yan, Jing Nathan Liu, Tianqi Chiu, Justin T Shen, Jiaming Qin, Zhen Yu, Yue Zhao, Yao Lakshmanan, Charu Kurzion, Yair Rush, Alexander M. Liu, Jialu Bendersky, Michael Computation and Language Comparative reasoning plays a crucial role in text preference prediction; however, large language models (LLMs) often demonstrate inconsistencies in their reasoning. While approaches like Chain-of-Thought improve accuracy in many other settings, they struggle to consistently distinguish the similarities and differences of complex texts. We introduce SC, a prompting approach that predicts text preferences by generating structured intermediate comparisons. SC begins by proposing aspects of comparison, followed by generating textual comparisons under each aspect. We select consistent comparisons with a pairwise consistency comparator that ensures each aspect's comparisons clearly distinguish differences between texts, significantly reducing hallucination and improving consistency. Our comprehensive evaluations across various NLP tasks, including summarization, retrieval, and automatic rating, demonstrate that SC equips LLMs to achieve state-of-the-art performance in text preference prediction. |
| title | Predicting Text Preference Via Structured Comparative Reasoning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.08390 |