No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917407943032832 |
|---|---|
| author | Li, Zhicong Jiang, Lingjie Hu, Yulan Zeng, Xingchen Li, Yixia Zhang, Xiangwen Chen, Guanhua Pan, Zheng Li, Xin Liu, Yong |
| author_facet | Li, Zhicong Jiang, Lingjie Hu, Yulan Zeng, Xingchen Li, Yixia Zhang, Xiangwen Chen, Guanhua Pan, Zheng Li, Xin Liu, Yong |
| contents | Critique-guided reinforcement learning (RL) has emerged as a powerful paradigm for training LLM agents by augmenting sparse outcome rewards with natural-language feedback. However, current methods often rely on static or offline critic models, which fail to adapt as the policy evolves. In on-policy RL, the agent's error patterns shift over time, causing stationary critics to become stale and providing feedback of diminishing utility. To address this, we introduce ECHO (Evolving Critic for Hindsight-Guided Optimization)}, a framework that jointly optimizes the policy and critic through a synchronized co-evolutionary loop. ECHO utilizes a cascaded rollout mechanism where the critic generates multiple diagnoses for an initial trajectory, followed by policy refinement to enable group-structured advantage estimation. We address the challenge of learning plateaus via a saturation-aware gain shaping objective, which rewards the critic for inducing incremental improvements in high-performing trajectories. By employing dual-track GRPO updates, ECHO ensures the critic's feedback stays synchronized with the evolving policy. Experimental results show that ECHO yields more stable training and higher long-horizon task success across open-world environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_06794 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning Li, Zhicong Jiang, Lingjie Hu, Yulan Zeng, Xingchen Li, Yixia Zhang, Xiangwen Chen, Guanhua Pan, Zheng Li, Xin Liu, Yong Artificial Intelligence Critique-guided reinforcement learning (RL) has emerged as a powerful paradigm for training LLM agents by augmenting sparse outcome rewards with natural-language feedback. However, current methods often rely on static or offline critic models, which fail to adapt as the policy evolves. In on-policy RL, the agent's error patterns shift over time, causing stationary critics to become stale and providing feedback of diminishing utility. To address this, we introduce ECHO (Evolving Critic for Hindsight-Guided Optimization)}, a framework that jointly optimizes the policy and critic through a synchronized co-evolutionary loop. ECHO utilizes a cascaded rollout mechanism where the critic generates multiple diagnoses for an initial trajectory, followed by policy refinement to enable group-structured advantage estimation. We address the challenge of learning plateaus via a saturation-aware gain shaping objective, which rewards the critic for inducing incremental improvements in high-performing trajectories. By employing dual-track GRPO updates, ECHO ensures the critic's feedback stays synchronized with the evolving policy. Experimental results show that ECHO yields more stable training and higher long-horizon task success across open-world environments. |
| title | No More Stale Feedback: Co-Evolving Critics for Open-World Agent Learning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2601.06794 |