MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910625255391232 |
|---|---|
| author | Zhang, Zeyu Dai, Quanyu Chen, Luyu Jiang, Zeren Li, Rui Zhu, Jieming Chen, Xu Xie, Yi Dong, Zhenhua Wen, Ji-Rong |
| author_facet | Zhang, Zeyu Dai, Quanyu Chen, Luyu Jiang, Zeren Li, Rui Zhu, Jieming Chen, Xu Xie, Yi Dong, Zhenhua Wen, Ji-Rong |
| contents | LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset. To benefit the research community, we have released our project at https://github.com/nuster1128/MemSim. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_20163 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants Zhang, Zeyu Dai, Quanyu Chen, Luyu Jiang, Zeren Li, Rui Zhu, Jieming Chen, Xu Xie, Yi Dong, Zhenhua Wen, Ji-Rong Artificial Intelligence Computation and Language LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset. To benefit the research community, we have released our project at https://github.com/nuster1128/MemSim. |
| title | MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2409.20163 |