MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908425084993536 |
|---|---|
| author | Tan, Haoran Zhang, Zeyu Ma, Chen Chen, Xu Dai, Quanyu Dong, Zhenhua |
| author_facet | Tan, Haoran Zhang, Zeyu Ma, Chen Chen, Xu Dai, Quanyu Dong, Zhenhua |
| contents | Recent works have highlighted the significance of memory mechanisms in LLM-based agents, which enable them to store observed information and adapt to dynamic environments. However, evaluating their memory capabilities still remains challenges. Previous evaluations are commonly limited by the diversity of memory levels and interactive scenarios. They also lack comprehensive metrics to reflect the memory capabilities from multiple aspects. To address these problems, in this paper, we construct a more comprehensive dataset and benchmark to evaluate the memory capability of LLM-based agents. Our dataset incorporates factual memory and reflective memory as different levels, and proposes participation and observation as various interactive scenarios. Based on our dataset, we present a benchmark, named MemBench, to evaluate the memory capability of LLM-based agents from multiple aspects, including their effectiveness, efficiency, and capacity. To benefit the research community, we release our dataset and project at https://github.com/import-myself/Membench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_21605 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents Tan, Haoran Zhang, Zeyu Ma, Chen Chen, Xu Dai, Quanyu Dong, Zhenhua Computation and Language Artificial Intelligence Recent works have highlighted the significance of memory mechanisms in LLM-based agents, which enable them to store observed information and adapt to dynamic environments. However, evaluating their memory capabilities still remains challenges. Previous evaluations are commonly limited by the diversity of memory levels and interactive scenarios. They also lack comprehensive metrics to reflect the memory capabilities from multiple aspects. To address these problems, in this paper, we construct a more comprehensive dataset and benchmark to evaluate the memory capability of LLM-based agents. Our dataset incorporates factual memory and reflective memory as different levels, and proposes participation and observation as various interactive scenarios. Based on our dataset, we present a benchmark, named MemBench, to evaluate the memory capability of LLM-based agents from multiple aspects, including their effectiveness, efficiency, and capacity. To benefit the research community, we release our dataset and project at https://github.com/import-myself/Membench. |
| title | MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.21605 |