Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2408.12799 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929692894822400 |
|---|---|
| author | Lee, JoonHo Son, JuYoun Seok, Juree Jang, Wooseok Kwon, Yeong-Dae |
| author_facet | Lee, JoonHo Son, JuYoun Seok, Juree Jang, Wooseok Kwon, Yeong-Dae |
| contents | Inconsistent annotations in training corpora, particularly within preference learning datasets, pose challenges in developing advanced language models. These inconsistencies often arise from variability among annotators and inherent multi-dimensional nature of the preferences. To address these issues, we introduce a self-curation method that preprocesses annotated datasets by leveraging proxy models trained directly on them. Our method enhances preference learning by automatically detecting and selecting consistent annotations. We validate the proposed approach through extensive instruction-following tasks, demonstrating performance improvements of up to 33\% across various learning algorithms and proxy capabilities. This work offers a straightforward and reliable solution to address preference inconsistencies without relying on heuristics, serving as an initial step toward the development of more advanced preference learning methodologies. Code is available at https://github.com/Self-Curation/ . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_12799 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora Lee, JoonHo Son, JuYoun Seok, Juree Jang, Wooseok Kwon, Yeong-Dae Computation and Language Artificial Intelligence Inconsistent annotations in training corpora, particularly within preference learning datasets, pose challenges in developing advanced language models. These inconsistencies often arise from variability among annotators and inherent multi-dimensional nature of the preferences. To address these issues, we introduce a self-curation method that preprocesses annotated datasets by leveraging proxy models trained directly on them. Our method enhances preference learning by automatically detecting and selecting consistent annotations. We validate the proposed approach through extensive instruction-following tasks, demonstrating performance improvements of up to 33\% across various learning algorithms and proxy capabilities. This work offers a straightforward and reliable solution to address preference inconsistencies without relying on heuristics, serving as an initial step toward the development of more advanced preference learning methodologies. Code is available at https://github.com/Self-Curation/ . |
| title | Preference Consistency Matters: Enhancing Preference Learning in Language Models with Automated Self-Curation of Training Corpora |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2408.12799 |