Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.16252 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918101391507456 |
|---|---|
| author | Nam, Hyunji Gottesman, Omer Zhang, Amy Foster, Dean Brunskill, Emma Ungar, Lyle |
| author_facet | Nam, Hyunji Gottesman, Omer Zhang, Amy Foster, Dean Brunskill, Emma Ungar, Lyle |
| contents | Large language models (LLMs) built on existing reinforcement learning with human feedback (RLHF) frameworks typically optimize responses based on immediate turn-level human preferences. However, this approach falls short in multi-turn dialogue settings, such as online math tutoring. We propose a method to enhance LLM-based tutors by representing the dialogue history with a lower-dimensional latent state representation of a student and optimizing a long-term policy to determine high-level actions based on the latent state. The goal is to better align the tutor's behavior with the long-term objective of guiding the student towards solving a target math problem on their own. Our model is lightweight, requiring less computational resources than prior work of training the tutor policy end-to-end to directly output the tutor's next utterance. Our experiment results demonstrate that these modifications lead to improved long-term outcomes compared to prompting in LLM-simulated tutoring tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_16252 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Efficient RL for optimizing conversation level outcomes with an LLM-based tutor Nam, Hyunji Gottesman, Omer Zhang, Amy Foster, Dean Brunskill, Emma Ungar, Lyle Computation and Language Artificial Intelligence Large language models (LLMs) built on existing reinforcement learning with human feedback (RLHF) frameworks typically optimize responses based on immediate turn-level human preferences. However, this approach falls short in multi-turn dialogue settings, such as online math tutoring. We propose a method to enhance LLM-based tutors by representing the dialogue history with a lower-dimensional latent state representation of a student and optimizing a long-term policy to determine high-level actions based on the latent state. The goal is to better align the tutor's behavior with the long-term objective of guiding the student towards solving a target math problem on their own. Our model is lightweight, requiring less computational resources than prior work of training the tutor policy end-to-end to directly output the tutor's next utterance. Our experiment results demonstrate that these modifications lead to improved long-term outcomes compared to prompting in LLM-simulated tutoring tasks. |
| title | Efficient RL for optimizing conversation level outcomes with an LLM-based tutor |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2507.16252 |