Training Language Models to Win Debates with Self-Play Improves Judge Accuracy
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910619274313728 |
|---|---|
| author | Arnesen, Samuel Rein, David Michael, Julian |
| author_facet | Arnesen, Samuel Rein, David Michael, Julian |
| contents | We test the robustness of debate as a method of scalable oversight by training models to debate with data generated via self-play. In a long-context reading comprehension task, we find that language model based evaluators answer questions more accurately when judging models optimized to win debates. By contrast, we find no such relationship for consultancy models trained to persuade a judge without an opposing debater present. In quantitative and qualitative comparisons between our debate models and novel consultancy baselines, we find evidence that debate training encourages stronger and more informative arguments, showing promise that it can help provide high-quality supervision for tasks that are difficult to directly evaluate. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_16636 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Training Language Models to Win Debates with Self-Play Improves Judge Accuracy Arnesen, Samuel Rein, David Michael, Julian Computation and Language Artificial Intelligence I.2.0; I.2.6 We test the robustness of debate as a method of scalable oversight by training models to debate with data generated via self-play. In a long-context reading comprehension task, we find that language model based evaluators answer questions more accurately when judging models optimized to win debates. By contrast, we find no such relationship for consultancy models trained to persuade a judge without an opposing debater present. In quantitative and qualitative comparisons between our debate models and novel consultancy baselines, we find evidence that debate training encourages stronger and more informative arguments, showing promise that it can help provide high-quality supervision for tasks that are difficult to directly evaluate. |
| title | Training Language Models to Win Debates with Self-Play Improves Judge Accuracy |
| topic | Computation and Language Artificial Intelligence I.2.0; I.2.6 |
| url | https://arxiv.org/abs/2409.16636 |