Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913828562796544 |
|---|---|
| author | Stöpler, Lennart Asadli, Rufat Nikolaus, Mitja Cotterell, Ryan Warstadt, Alex |
| author_facet | Stöpler, Lennart Asadli, Rufat Nikolaus, Mitja Cotterell, Ryan Warstadt, Alex |
| contents | We propose a method for training language models in an interactive setting inspired by child language acquisition. In our setting, a speaker attempts to communicate some information to a listener in a single-turn dialogue and receives a reward if communicative success is achieved. Unlike earlier related work using image--caption data for interactive reference games, we operationalize communicative success in a more abstract language-only question--answering setting. First, we present a feasibility study demonstrating that our reward provides an indirect signal about grammaticality. Second, we conduct experiments using reinforcement learning to fine-tune language models. We observe that cognitively plausible constraints on the communication channel lead to interpretable changes in speaker behavior. However, we do not yet see improvements on linguistic evaluations from our training regime. We outline potential modifications to the task design and training configuration that could better position future work to use our methodology to observe the benefits of interaction on language learning in computational cognitive models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_05970 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models Stöpler, Lennart Asadli, Rufat Nikolaus, Mitja Cotterell, Ryan Warstadt, Alex Computation and Language We propose a method for training language models in an interactive setting inspired by child language acquisition. In our setting, a speaker attempts to communicate some information to a listener in a single-turn dialogue and receives a reward if communicative success is achieved. Unlike earlier related work using image--caption data for interactive reference games, we operationalize communicative success in a more abstract language-only question--answering setting. First, we present a feasibility study demonstrating that our reward provides an indirect signal about grammaticality. Second, we conduct experiments using reinforcement learning to fine-tune language models. We observe that cognitively plausible constraints on the communication channel lead to interpretable changes in speaker behavior. However, we do not yet see improvements on linguistic evaluations from our training regime. We outline potential modifications to the task design and training configuration that could better position future work to use our methodology to observe the benefits of interaction on language learning in computational cognitive models. |
| title | Towards Developmentally Plausible Rewards: Communicative Success as a Learning Signal for Interactive Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.05970 |