Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914891749654528 |
|---|---|
| author | Wu, Tianhao Yuan, Weizhe Golovneva, Olga Xu, Jing Tian, Yuandong Jiao, Jiantao Weston, Jason Sukhbaatar, Sainbayar |
| author_facet | Wu, Tianhao Yuan, Weizhe Golovneva, Olga Xu, Jing Tian, Yuandong Jiao, Jiantao Weston, Jason Sukhbaatar, Sainbayar |
| contents | Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs can improve by judging their own responses instead of relying on human labelers. However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills. Surprisingly, this unsupervised approach improves the model's ability to judge {\em and} follow instructions, as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard. These results strongly suggest the potential for self-improving models without human supervision. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_19594 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge Wu, Tianhao Yuan, Weizhe Golovneva, Olga Xu, Jing Tian, Yuandong Jiao, Jiantao Weston, Jason Sukhbaatar, Sainbayar Computation and Language Artificial Intelligence Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024) have shown that LLMs can improve by judging their own responses instead of relying on human labelers. However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills. Surprisingly, this unsupervised approach improves the model's ability to judge {\em and} follow instructions, as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard. These results strongly suggest the potential for self-improving models without human supervision. |
| title | Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.19594 |