Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913436815851520 |
|---|---|
| author | Bailis, Suma Friedhoff, Jane Chen, Feiyang |
| author_facet | Bailis, Suma Friedhoff, Jane Chen, Feiyang |
| contents | This paper introduces Werewolf Arena, a novel framework for evaluating large language models (LLMs) through the lens of the classic social deduction game, Werewolf. In Werewolf Arena, LLMs compete against each other, navigating the game's complex dynamics of deception, deduction, and persuasion. The framework introduces a dynamic turn-taking system based on bidding, mirroring real-world discussions where individuals strategically choose when to speak. We demonstrate the framework's utility through an arena-style tournament featuring Gemini and GPT models. Our results reveal distinct strengths and weaknesses in the models' strategic reasoning and communication. These findings highlight Werewolf Arena's potential as a challenging and scalable LLM benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_13943 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction Bailis, Suma Friedhoff, Jane Chen, Feiyang Computation and Language Artificial Intelligence This paper introduces Werewolf Arena, a novel framework for evaluating large language models (LLMs) through the lens of the classic social deduction game, Werewolf. In Werewolf Arena, LLMs compete against each other, navigating the game's complex dynamics of deception, deduction, and persuasion. The framework introduces a dynamic turn-taking system based on bidding, mirroring real-world discussions where individuals strategically choose when to speak. We demonstrate the framework's utility through an arena-style tournament featuring Gemini and GPT models. Our results reveal distinct strengths and weaknesses in the models' strategic reasoning and communication. These findings highlight Werewolf Arena's potential as a challenging and scalable LLM benchmark. |
| title | Werewolf Arena: A Case Study in LLM Evaluation via Social Deduction |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.13943 |