MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913125532434432 |
|---|---|
| author | Liao, Junwei Shi, Haoting Zhou, Ruiwen Wang, Jiaqian Zhang, Shengtao Zhang, Wei Wen, Ying Li, Zhiyu Xiong, Feiyu Tang, Bo Zhang, Weinan Wen, Muning |
| author_facet | Liao, Junwei Shi, Haoting Zhou, Ruiwen Wang, Jiaqian Zhang, Shengtao Zhang, Wei Wen, Ying Li, Zhiyu Xiong, Feiyu Tang, Bo Zhang, Weinan Wen, Muning |
| contents | Episodic memory allows LLM agents to accumulate and retrieve experience, but current methods treat each memory independently, i.e., evaluating retrieval quality in isolation without accounting for the dependency chains through which memories enable the creation of future memories. We introduce MemQ, which applies TD($λ$) eligibility traces to memory Q-values, propagating credit backward through a provenance DAG that records which memories were retrieved when each new memory was created. Credit weight decays as $(γλ)^d$ with DAG depth $d$, replacing temporal distance with structural proximity. We formalize the setting as an Exogenous-Context MDP, whose factored transition decouples the exogenous task stream from the endogenous memory store. Across six benchmarks, spanning OS interaction, function calling, code generation, multimodal reasoning, embodied reasoning, and expert-level QA, MemQ achieves the highest success rate on all six in generalization evaluation and runtime learning, with gains largest on multi-step tasks that produce deep and relevant provenance chains (up to +5.7~pp) and smallest on single-step classification (+0.77~pp) where single-step updates already suffice. We further study how $γ$ and $λ$ interact with the EC-MDP structure, providing principled guidance for parameter selection and future research. Code is available at https://github.com/jwliao-ai/MemQ. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_08374 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs Liao, Junwei Shi, Haoting Zhou, Ruiwen Wang, Jiaqian Zhang, Shengtao Zhang, Wei Wen, Ying Li, Zhiyu Xiong, Feiyu Tang, Bo Zhang, Weinan Wen, Muning Artificial Intelligence Episodic memory allows LLM agents to accumulate and retrieve experience, but current methods treat each memory independently, i.e., evaluating retrieval quality in isolation without accounting for the dependency chains through which memories enable the creation of future memories. We introduce MemQ, which applies TD($λ$) eligibility traces to memory Q-values, propagating credit backward through a provenance DAG that records which memories were retrieved when each new memory was created. Credit weight decays as $(γλ)^d$ with DAG depth $d$, replacing temporal distance with structural proximity. We formalize the setting as an Exogenous-Context MDP, whose factored transition decouples the exogenous task stream from the endogenous memory store. Across six benchmarks, spanning OS interaction, function calling, code generation, multimodal reasoning, embodied reasoning, and expert-level QA, MemQ achieves the highest success rate on all six in generalization evaluation and runtime learning, with gains largest on multi-step tasks that produce deep and relevant provenance chains (up to +5.7~pp) and smallest on single-step classification (+0.77~pp) where single-step updates already suffice. We further study how $γ$ and $λ$ interact with the EC-MDP structure, providing principled guidance for parameter selection and future research. Code is available at https://github.com/jwliao-ai/MemQ. |
| title | MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2605.08374 |