Cover Image

Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Smith, Chandler, Abdulhai, Marwa, Diaz, Manfred, Tesic, Marko, Trivedi, Rakshit S., Vezhnevets, Alexander Sasha, Hammond, Lewis, Clifton, Jesse, Chang, Minsuk, Duéñez-Guzmán, Edgar A., Agapiou, John P., Matyas, Jayd, Karmon, Danny, Kundu, Akash, Korshuk, Aliaksei, Ananya, Ananya, Rahman, Arrasy, Kulandaivel, Avinaash Anand, McHale, Bain, Zhang, Beining, Alexander, Buyantuev, Rojas, Carlos Saith Rodriguez, Wang, Caroline, Talele, Chetan, Liu, Chenao, Lin, Chichen, Riazi, Diana, Shi, Di Yang, Tewolde, Emanuel, Tennant, Elizaveta, Zhong, Fangwei, Cui, Fuyang, Zhao, Gang, Piqueras, Gema Parreño, Yun, Hyeonggeun, Makarov, Ilya, Cui, Jiaxun, Purbey, Jebish, Dilkes, Jim, Nguyen, Jord, Xiao, Lingyun, Giraldo, Luis Felipe, Chacon-Chamorro, Manuela, Beltran, Manuel Sebastian Rios, Segura, Marta Emili García, Wang, Mengmeng, Alim, Mogtaba, Quijano, Nicanor, Schiavone, Nico, Macmillan-Scott, Olivia, Peña, Oswaldo, Stone, Peter, Kadiyala, Ram Mohan Rao, Fernandez, Rolando, Manrique, Ruben, Lu, Sunjia, McIlraith, Sheila A., Dhuri, Shamika, Shi, Shuqing, Gupta, Siddhant, Sarangi, Sneheel, Subramanian, Sriram Ganapathi, Cha, Taehun, Klassen, Toryn Q., Tu, Wenming, Fan, Weijian, Ruiyang, Wu, Feng, Xue, Du, Yali, Liu, Yang, Wang, Yiding, Kang, Yipeng, Sung, Yoonchang, Chen, Yuxuan, Zhang, Zhaowei, Wang, Zhihan, Wu, Zhiqiang, Chen, Ziang, Zheng, Zilong, Jia, Zixia, Wang, Ziyan, Hadfield-Menell, Dylan, Jaques, Natasha, Baarslag, Tim, Hernandez-Orallo, Jose, Leibo, Joel Z.
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence 68T42 I.2.6
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866911299615588352
author	Smith, Chandler Abdulhai, Marwa Diaz, Manfred Tesic, Marko Trivedi, Rakshit S. Vezhnevets, Alexander Sasha Hammond, Lewis Clifton, Jesse Chang, Minsuk Duéñez-Guzmán, Edgar A. Agapiou, John P. Matyas, Jayd Karmon, Danny Kundu, Akash Korshuk, Aliaksei Ananya, Ananya Rahman, Arrasy Kulandaivel, Avinaash Anand McHale, Bain Zhang, Beining Alexander, Buyantuev Rojas, Carlos Saith Rodriguez Wang, Caroline Talele, Chetan Liu, Chenao Lin, Chichen Riazi, Diana Shi, Di Yang Tewolde, Emanuel Tennant, Elizaveta Zhong, Fangwei Cui, Fuyang Zhao, Gang Piqueras, Gema Parreño Yun, Hyeonggeun Makarov, Ilya Cui, Jiaxun Purbey, Jebish Dilkes, Jim Nguyen, Jord Xiao, Lingyun Giraldo, Luis Felipe Chacon-Chamorro, Manuela Beltran, Manuel Sebastian Rios Segura, Marta Emili García Wang, Mengmeng Alim, Mogtaba Quijano, Nicanor Schiavone, Nico Macmillan-Scott, Olivia Peña, Oswaldo Stone, Peter Kadiyala, Ram Mohan Rao Fernandez, Rolando Manrique, Ruben Lu, Sunjia McIlraith, Sheila A. Dhuri, Shamika Shi, Shuqing Gupta, Siddhant Sarangi, Sneheel Subramanian, Sriram Ganapathi Cha, Taehun Klassen, Toryn Q. Tu, Wenming Fan, Weijian Ruiyang, Wu Feng, Xue Du, Yali Liu, Yang Wang, Yiding Kang, Yipeng Sung, Yoonchang Chen, Yuxuan Zhang, Zhaowei Wang, Zhihan Wu, Zhiqiang Chen, Ziang Zheng, Zilong Jia, Zixia Wang, Ziyan Hadfield-Menell, Dylan Jaques, Natasha Baarslag, Tim Hernandez-Orallo, Jose Leibo, Joel Z.
author_facet	Smith, Chandler Abdulhai, Marwa Diaz, Manfred Tesic, Marko Trivedi, Rakshit S. Vezhnevets, Alexander Sasha Hammond, Lewis Clifton, Jesse Chang, Minsuk Duéñez-Guzmán, Edgar A. Agapiou, John P. Matyas, Jayd Karmon, Danny Kundu, Akash Korshuk, Aliaksei Ananya, Ananya Rahman, Arrasy Kulandaivel, Avinaash Anand McHale, Bain Zhang, Beining Alexander, Buyantuev Rojas, Carlos Saith Rodriguez Wang, Caroline Talele, Chetan Liu, Chenao Lin, Chichen Riazi, Diana Shi, Di Yang Tewolde, Emanuel Tennant, Elizaveta Zhong, Fangwei Cui, Fuyang Zhao, Gang Piqueras, Gema Parreño Yun, Hyeonggeun Makarov, Ilya Cui, Jiaxun Purbey, Jebish Dilkes, Jim Nguyen, Jord Xiao, Lingyun Giraldo, Luis Felipe Chacon-Chamorro, Manuela Beltran, Manuel Sebastian Rios Segura, Marta Emili García Wang, Mengmeng Alim, Mogtaba Quijano, Nicanor Schiavone, Nico Macmillan-Scott, Olivia Peña, Oswaldo Stone, Peter Kadiyala, Ram Mohan Rao Fernandez, Rolando Manrique, Ruben Lu, Sunjia McIlraith, Sheila A. Dhuri, Shamika Shi, Shuqing Gupta, Siddhant Sarangi, Sneheel Subramanian, Sriram Ganapathi Cha, Taehun Klassen, Toryn Q. Tu, Wenming Fan, Weijian Ruiyang, Wu Feng, Xue Du, Yali Liu, Yang Wang, Yiding Kang, Yipeng Sung, Yoonchang Chen, Yuxuan Zhang, Zhaowei Wang, Zhihan Wu, Zhiqiang Chen, Ziang Zheng, Zilong Jia, Zixia Wang, Ziyan Hadfield-Menell, Dylan Jaques, Natasha Baarslag, Tim Hernandez-Orallo, Jose Leibo, Joel Z.
contents	Large Language Model (LLM) agents have demonstrated impressive capabilities for social interaction and are increasingly being deployed in situations where they might engage with both human and artificial agents. These interactions represent a critical frontier for LLM-based agents, yet existing evaluation methods fail to measure how well these capabilities generalize to novel social situations. In this paper, we introduce a method for evaluating the ability of LLM-based agents to cooperate in zero-shot, mixed-motive environments using Concordia, a natural language multi-agent simulation environment. Our method measures general cooperative intelligence by testing an agent's ability to identify and exploit opportunities for mutual gain across diverse partners and contexts. We present empirical results from the NeurIPS 2024 Concordia Contest, where agents were evaluated on their ability to achieve mutual gains across a suite of diverse scenarios ranging from negotiation to collective action problems. Our findings reveal significant gaps between current agent capabilities and the robust generalization required for reliable cooperation, particularly in scenarios demanding persuasion and norm enforcement.
format	Preprint
id	arxiv_https___arxiv_org_abs_2512_03318
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia Smith, Chandler Abdulhai, Marwa Diaz, Manfred Tesic, Marko Trivedi, Rakshit S. Vezhnevets, Alexander Sasha Hammond, Lewis Clifton, Jesse Chang, Minsuk Duéñez-Guzmán, Edgar A. Agapiou, John P. Matyas, Jayd Karmon, Danny Kundu, Akash Korshuk, Aliaksei Ananya, Ananya Rahman, Arrasy Kulandaivel, Avinaash Anand McHale, Bain Zhang, Beining Alexander, Buyantuev Rojas, Carlos Saith Rodriguez Wang, Caroline Talele, Chetan Liu, Chenao Lin, Chichen Riazi, Diana Shi, Di Yang Tewolde, Emanuel Tennant, Elizaveta Zhong, Fangwei Cui, Fuyang Zhao, Gang Piqueras, Gema Parreño Yun, Hyeonggeun Makarov, Ilya Cui, Jiaxun Purbey, Jebish Dilkes, Jim Nguyen, Jord Xiao, Lingyun Giraldo, Luis Felipe Chacon-Chamorro, Manuela Beltran, Manuel Sebastian Rios Segura, Marta Emili García Wang, Mengmeng Alim, Mogtaba Quijano, Nicanor Schiavone, Nico Macmillan-Scott, Olivia Peña, Oswaldo Stone, Peter Kadiyala, Ram Mohan Rao Fernandez, Rolando Manrique, Ruben Lu, Sunjia McIlraith, Sheila A. Dhuri, Shamika Shi, Shuqing Gupta, Siddhant Sarangi, Sneheel Subramanian, Sriram Ganapathi Cha, Taehun Klassen, Toryn Q. Tu, Wenming Fan, Weijian Ruiyang, Wu Feng, Xue Du, Yali Liu, Yang Wang, Yiding Kang, Yipeng Sung, Yoonchang Chen, Yuxuan Zhang, Zhaowei Wang, Zhihan Wu, Zhiqiang Chen, Ziang Zheng, Zilong Jia, Zixia Wang, Ziyan Hadfield-Menell, Dylan Jaques, Natasha Baarslag, Tim Hernandez-Orallo, Jose Leibo, Joel Z. Artificial Intelligence 68T42 I.2.6 Large Language Model (LLM) agents have demonstrated impressive capabilities for social interaction and are increasingly being deployed in situations where they might engage with both human and artificial agents. These interactions represent a critical frontier for LLM-based agents, yet existing evaluation methods fail to measure how well these capabilities generalize to novel social situations. In this paper, we introduce a method for evaluating the ability of LLM-based agents to cooperate in zero-shot, mixed-motive environments using Concordia, a natural language multi-agent simulation environment. Our method measures general cooperative intelligence by testing an agent's ability to identify and exploit opportunities for mutual gain across diverse partners and contexts. We present empirical results from the NeurIPS 2024 Concordia Contest, where agents were evaluated on their ability to achieve mutual gains across a suite of diverse scenarios ranging from negotiation to collective action problems. Our findings reveal significant gaps between current agent capabilities and the robust generalization required for reliable cooperation, particularly in scenarios demanding persuasion and norm enforcement.
title	Evaluating Generalization Capabilities of LLM-Based Agents in Mixed-Motive Scenarios Using Concordia
topic	Artificial Intelligence 68T42 I.2.6
url	https://arxiv.org/abs/2512.03318