Language Alignment via Nash-learning and Adaptive feedback
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929395699023872 |
|---|---|
| author | Azarafrooz, Ari Faal, Farshid |
| author_facet | Azarafrooz, Ari Faal, Farshid |
| contents | Recent research has shown the potential of Nash Learning via Human Feedback for large language model alignment by incorporating the notion of a preference model in a minimax game setup. We take this idea further by casting the alignment as a mirror descent algorithm against the adaptive feedback of an improved opponent, thereby removing the need for learning a preference model or the existence of an annotated dataset altogether. The resulting algorithm, which we refer to as Language Alignment via Nash-learning and Adaptive feedback (LANA), is capable of self-alignment without the need for a human-annotated preference dataset. We support this statement with various experiments and mathematical discussion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_15890 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Language Alignment via Nash-learning and Adaptive feedback Azarafrooz, Ari Faal, Farshid Machine Learning Artificial Intelligence Computation and Language Computer Science and Game Theory Recent research has shown the potential of Nash Learning via Human Feedback for large language model alignment by incorporating the notion of a preference model in a minimax game setup. We take this idea further by casting the alignment as a mirror descent algorithm against the adaptive feedback of an improved opponent, thereby removing the need for learning a preference model or the existence of an annotated dataset altogether. The resulting algorithm, which we refer to as Language Alignment via Nash-learning and Adaptive feedback (LANA), is capable of self-alignment without the need for a human-annotated preference dataset. We support this statement with various experiments and mathematical discussion. |
| title | Language Alignment via Nash-learning and Adaptive feedback |
| topic | Machine Learning Artificial Intelligence Computation and Language Computer Science and Game Theory |
| url | https://arxiv.org/abs/2406.15890 |