Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.06194 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910191399731200 |
|---|---|
| author | Zhang, Naifan Sun, Ruihan Su, Jinwei Yang, Hengjie Pan, Zhengyuan Chen, Zhaohan Zhang, Xiaofan |
| author_facet | Zhang, Naifan Sun, Ruihan Su, Jinwei Yang, Hengjie Pan, Zhengyuan Chen, Zhaohan Zhang, Xiaofan |
| contents | Reinforcement learning (RL) for large language models (LLMs) has shown strong performance in single-turn tasks, but extending it to multi-turn interaction remains challenging due to sparse rewards and poor per-turn credit assignment. In emotional support dialogues, responses shape future user states, so matched-state step-wise comparison is unavailable, while trajectory-level supervision is insufficient. We propose MICA (Multi-granularity Intertemporal Credit Assignment), a critic-free RL framework for multi-turn emotional support tasks. MICA derives both immediate and delayed credit from a shared potential function over the user's structured support state. Incremental Distance Reward measures the per-turn decrease in residual distance to the target state, while its Monte Carlo return captures delayed effects. After scope-specific normalization, the two signals form a mixed advantage for stable per-turn optimization without matched-state comparisons, rollout trees, or a learned critic. On EMPA, EQ-Bench, and EmoBench with Qwen2.5-7B-Instruct and Qwen3-8B/14B/32B, MICA consistently outperforms GRPO and REINFORCE++, achieving up to +43.2 on EMPA, while adding no rollout cost and remaining robust to reward judges. These results show that turn-aware credit assignment enables effective and practical multi-turn RL for interactive LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_06194 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue Zhang, Naifan Sun, Ruihan Su, Jinwei Yang, Hengjie Pan, Zhengyuan Chen, Zhaohan Zhang, Xiaofan Computation and Language Artificial Intelligence Reinforcement learning (RL) for large language models (LLMs) has shown strong performance in single-turn tasks, but extending it to multi-turn interaction remains challenging due to sparse rewards and poor per-turn credit assignment. In emotional support dialogues, responses shape future user states, so matched-state step-wise comparison is unavailable, while trajectory-level supervision is insufficient. We propose MICA (Multi-granularity Intertemporal Credit Assignment), a critic-free RL framework for multi-turn emotional support tasks. MICA derives both immediate and delayed credit from a shared potential function over the user's structured support state. Incremental Distance Reward measures the per-turn decrease in residual distance to the target state, while its Monte Carlo return captures delayed effects. After scope-specific normalization, the two signals form a mixed advantage for stable per-turn optimization without matched-state comparisons, rollout trees, or a learned critic. On EMPA, EQ-Bench, and EmoBench with Qwen2.5-7B-Instruct and Qwen3-8B/14B/32B, MICA consistently outperforms GRPO and REINFORCE++, achieving up to +43.2 on EMPA, while adding no rollout cost and remaining robust to reward judges. These results show that turn-aware credit assignment enables effective and practical multi-turn RL for interactive LLMs. |
| title | MICA: Multi-granularity Intertemporal Credit Assignment for Long-Horizon Emotional Support Dialogue |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2603.06194 |