Evaluating Language Model Agency through Negotiations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915802674888704 |
|---|---|
| author | Davidson, Tim R. Veselovsky, Veniamin Josifoski, Martin Peyrard, Maxime Bosselut, Antoine Kosinski, Michal West, Robert |
| author_facet | Davidson, Tim R. Veselovsky, Veniamin Josifoski, Martin Peyrard, Maxime Bosselut, Antoine Kosinski, Michal West, Robert |
| contents | We introduce an approach to evaluate language model (LM) agency using negotiation games. This approach better reflects real-world use cases and addresses some of the shortcomings of alternative LM benchmarks. Negotiation games enable us to study multi-turn, and cross-model interactions, modulate complexity, and side-step accidental evaluation data leakage. We use our approach to test six widely used and publicly accessible LMs, evaluating performance and alignment in both self-play and cross-play settings. Noteworthy findings include: (i) only closed-source models tested here were able to complete these tasks; (ii) cooperative bargaining games proved to be most challenging to the models; and (iii) even the most powerful models sometimes "lose" to weaker opponents |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_04536 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating Language Model Agency through Negotiations Davidson, Tim R. Veselovsky, Veniamin Josifoski, Martin Peyrard, Maxime Bosselut, Antoine Kosinski, Michal West, Robert Computation and Language Artificial Intelligence Machine Learning We introduce an approach to evaluate language model (LM) agency using negotiation games. This approach better reflects real-world use cases and addresses some of the shortcomings of alternative LM benchmarks. Negotiation games enable us to study multi-turn, and cross-model interactions, modulate complexity, and side-step accidental evaluation data leakage. We use our approach to test six widely used and publicly accessible LMs, evaluating performance and alignment in both self-play and cross-play settings. Noteworthy findings include: (i) only closed-source models tested here were able to complete these tasks; (ii) cooperative bargaining games proved to be most challenging to the models; and (iii) even the most powerful models sometimes "lose" to weaker opponents |
| title | Evaluating Language Model Agency through Negotiations |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2401.04536 |