Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908788873756672 |
|---|---|
| author | Li, Yibo Lin, Zijie Deng, Ailin Zhang, Xuan He, Yufei Ji, Shuo Cao, Tri Hooi, Bryan |
| author_facet | Li, Yibo Lin, Zijie Deng, Ailin Zhang, Xuan He, Yufei Ji, Shuo Cao, Tri Hooi, Bryan |
| contents | While Large Language Model (LLM) agents excel at general tasks, they inherently struggle with continual adaptation due to the frozen weights after deployment. Conventional reinforcement learning (RL) offers a solution but incurs prohibitive computational costs and the risk of catastrophic forgetting. We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates. JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly. These estimates are then used to directly modulate the LLM's output logits. We theoretically prove that this additive update rule is the exact closed-form solution to the KL-constrained policy optimization objective. Extensive experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods. Crucially, JitRL outperforms the performance of computationally expensive fine-tuning methods (e.g., WebRL) while reducing monetary costs by over 30 times, offering a scalable path for continual learning agents. The code is available at https://github.com/liushiliushi/JitRL. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_18510 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates Li, Yibo Lin, Zijie Deng, Ailin Zhang, Xuan He, Yufei Ji, Shuo Cao, Tri Hooi, Bryan Machine Learning Artificial Intelligence While Large Language Model (LLM) agents excel at general tasks, they inherently struggle with continual adaptation due to the frozen weights after deployment. Conventional reinforcement learning (RL) offers a solution but incurs prohibitive computational costs and the risk of catastrophic forgetting. We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates. JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly. These estimates are then used to directly modulate the LLM's output logits. We theoretically prove that this additive update rule is the exact closed-form solution to the KL-constrained policy optimization objective. Extensive experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods. Crucially, JitRL outperforms the performance of computationally expensive fine-tuning methods (e.g., WebRL) while reducing monetary costs by over 30 times, offering a scalable path for continual learning agents. The code is available at https://github.com/liushiliushi/JitRL. |
| title | Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2601.18510 |