Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.08590 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916966268141568 |
|---|---|
| author | Horst, Nicola Mazzaccara, Davide Schmidt, Antonia Sullivan, Michael Momentè, Filippo Franceschetti, Luca Sadler, Philipp Hakimov, Sherzod Testoni, Alberto Bernardi, Raffaella Fernández, Raquel Koller, Alexander Lemon, Oliver Schlangen, David Giulianelli, Mario Suglia, Alessandro |
| author_facet | Horst, Nicola Mazzaccara, Davide Schmidt, Antonia Sullivan, Michael Momentè, Filippo Franceschetti, Luca Sadler, Philipp Hakimov, Sherzod Testoni, Alberto Bernardi, Raffaella Fernández, Raquel Koller, Alexander Lemon, Oliver Schlangen, David Giulianelli, Mario Suglia, Alessandro |
| contents | Interaction between learner and feedback-giver has come into focus recently for post-training of Large Language Models (LLMs), through the use of reward models that judge the appropriateness of a model's response. In this paper, we investigate whether Dialogue Games -- goal-directed and rule-governed activities driven predominantly by verbal actions -- can also serve as a source of feedback signals for learning. We introduce Playpen, an environment for off- and online learning through Dialogue Game self-play, and investigate a representative set of post-training methods: supervised fine-tuning; direct alignment (DPO); and reinforcement learning with GRPO. We experiment with post-training a small LLM (Llama-3.1-8B-Instruct), evaluating performance on unseen instances of training games as well as unseen games, and on standard benchmarks. We find that imitation learning through SFT improves performance on unseen instances, but negatively impacts other skills, while interactive learning with GRPO shows balanced improvements without loss of skills. We release the framework and the baseline training setups to foster research in the promising new direction of learning in (synthetic) interaction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_08590 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Playpen: An Environment for Exploring Learning Through Conversational Interaction Horst, Nicola Mazzaccara, Davide Schmidt, Antonia Sullivan, Michael Momentè, Filippo Franceschetti, Luca Sadler, Philipp Hakimov, Sherzod Testoni, Alberto Bernardi, Raffaella Fernández, Raquel Koller, Alexander Lemon, Oliver Schlangen, David Giulianelli, Mario Suglia, Alessandro Computation and Language Interaction between learner and feedback-giver has come into focus recently for post-training of Large Language Models (LLMs), through the use of reward models that judge the appropriateness of a model's response. In this paper, we investigate whether Dialogue Games -- goal-directed and rule-governed activities driven predominantly by verbal actions -- can also serve as a source of feedback signals for learning. We introduce Playpen, an environment for off- and online learning through Dialogue Game self-play, and investigate a representative set of post-training methods: supervised fine-tuning; direct alignment (DPO); and reinforcement learning with GRPO. We experiment with post-training a small LLM (Llama-3.1-8B-Instruct), evaluating performance on unseen instances of training games as well as unseen games, and on standard benchmarks. We find that imitation learning through SFT improves performance on unseen instances, but negatively impacts other skills, while interactive learning with GRPO shows balanced improvements without loss of skills. We release the framework and the baseline training setups to foster research in the promising new direction of learning in (synthetic) interaction. |
| title | Playpen: An Environment for Exploring Learning Through Conversational Interaction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.08590 |