What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911697227218944 |
|---|---|
| author | Li, Xiaozhe Lyu, Tianyi Li, Yang Ma, Yichuan Li, Peiji Li, Linyang Guo, Qipeng Lin, Dahua Chen, Kai |
| author_facet | Li, Xiaozhe Lyu, Tianyi Li, Yang Ma, Yichuan Li, Peiji Li, Linyang Guo, Qipeng Lin, Dahua Chen, Kai |
| contents | Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging: a single success-or-failure signal must be distributed across many actions. Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback. Multi-turn agent settings are underexplored, where feedback can include error messages, page changes, observations, or reference trajectories. We systematically study five feedback sources and two insertion granularities and introduce SERL, a selective environment-reweighted learning framework. SERL uses the task reward to determine update direction, while environment feedback adjusts placement and magnitude, focusing on critical actions. On ALFWorld and WebShop, SERL achieves 90.0% and 80.1% success, outperforming strong RL and distillation baselines. Analysis shows that grounded, action-relevant feedback at meaningful points consistently outperforms indiscriminate use of longer or richer context. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_19447 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents Li, Xiaozhe Lyu, Tianyi Li, Yang Ma, Yichuan Li, Peiji Li, Linyang Guo, Qipeng Lin, Dahua Chen, Kai Artificial Intelligence Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging: a single success-or-failure signal must be distributed across many actions. Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback. Multi-turn agent settings are underexplored, where feedback can include error messages, page changes, observations, or reference trajectories. We systematically study five feedback sources and two insertion granularities and introduce SERL, a selective environment-reweighted learning framework. SERL uses the task reward to determine update direction, while environment feedback adjusts placement and magnitude, focusing on critical actions. On ALFWorld and WebShop, SERL achieves 90.0% and 80.1% success, outperforming strong RL and distillation baselines. Analysis shows that grounded, action-relevant feedback at meaningful points consistently outperforms indiscriminate use of longer or richer context. |
| title | What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2605.19447 |