Multicultural Spyfall: Assessing LLMs through Dynamic Multilingual Social Deduction Game
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918288236216320 |
|---|---|
| author | Wibowo, Haryo Akbarianto Elsetohy, Alaa Cui, Qinrong Aji, Alham Fikri |
| author_facet | Wibowo, Haryo Akbarianto Elsetohy, Alaa Cui, Qinrong Aji, Alham Fikri |
| contents | The rapid advancement of Large Language Models (LLMs) has necessitated more robust evaluation methods that go beyond static benchmarks, which are increasingly prone to data saturation and leakage. In this paper, we propose a dynamic benchmarking framework for evaluating multilingual and multicultural capabilities through the social deduction game Spyfall. In our setup, models must engage in strategic dialogue to either identify a secret agent or avoid detection, utilizing culturally relevant locations or local foods. Our results show that our game-based rankings align closely with the Chatbot Arena. However, we find a significant performance gap in non-English contexts: models are generally less proficient when handling locally specific entities and often struggle with rule-following or strategic integrity in non-English languages. We demonstrate that this game-based approach provides a scalable, leakage-resistant, and culturally nuanced alternative to traditional NLP benchmarks. The game history can be accessed here https://huggingface.co/datasets/haryoaw/cultural-spyfall. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_09017 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Multicultural Spyfall: Assessing LLMs through Dynamic Multilingual Social Deduction Game Wibowo, Haryo Akbarianto Elsetohy, Alaa Cui, Qinrong Aji, Alham Fikri Computation and Language 68T50 The rapid advancement of Large Language Models (LLMs) has necessitated more robust evaluation methods that go beyond static benchmarks, which are increasingly prone to data saturation and leakage. In this paper, we propose a dynamic benchmarking framework for evaluating multilingual and multicultural capabilities through the social deduction game Spyfall. In our setup, models must engage in strategic dialogue to either identify a secret agent or avoid detection, utilizing culturally relevant locations or local foods. Our results show that our game-based rankings align closely with the Chatbot Arena. However, we find a significant performance gap in non-English contexts: models are generally less proficient when handling locally specific entities and often struggle with rule-following or strategic integrity in non-English languages. We demonstrate that this game-based approach provides a scalable, leakage-resistant, and culturally nuanced alternative to traditional NLP benchmarks. The game history can be accessed here https://huggingface.co/datasets/haryoaw/cultural-spyfall. |
| title | Multicultural Spyfall: Assessing LLMs through Dynamic Multilingual Social Deduction Game |
| topic | Computation and Language 68T50 |
| url | https://arxiv.org/abs/2601.09017 |