EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915763147767808 |
|---|---|
| author | Yao, Yiqun Yu, Naitong Li, Xiang Jiang, Xin Fang, Xuezhi Ma, Wenjia Meng, Xuying Li, Jing Sun, Aixin Wang, Yequan |
| author_facet | Yao, Yiqun Yu, Naitong Li, Xiang Jiang, Xin Fang, Xuezhi Ma, Wenjia Meng, Xuying Li, Jing Sun, Aixin Wang, Yequan |
| contents | We introduce EgoMem, the first lifelong memory agent tailored for full-duplex models that process real-time omnimodal streams. EgoMem enables real-time models to recognize multiple users directly from raw audiovisual streams, to provide personalized response, and to maintain long-term knowledge of users' facts, preferences, and social relationships extracted from audiovisual history. EgoMem operates with three asynchronous processes: (i) a retrieval process that dynamically identifies user via face and voice, and gathers relevant context from a long-term memory; (ii) an omnimodal dialog process that generates personalized audio responses based on the retrieved context; and (iii) a memory management process that automatically detects dialog boundaries from omnimodal streams, and extracts necessary information to update the long-term memory. Unlike existing memory agents for LLMs, EgoMem relies entirely on raw audiovisual streams, making it especially suitable for lifelong, real-time, and embodied scenarios. Experimental results demonstrate that EgoMem's retrieval and memory management modules achieve over 95% accuracy on the test set. When integrated with a fine-tuned RoboEgo omnimodal chatbot, the system achieves fact-consistency scores above 87% in real-time personalized dialogs, establishing a strong baseline for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_11914 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models Yao, Yiqun Yu, Naitong Li, Xiang Jiang, Xin Fang, Xuezhi Ma, Wenjia Meng, Xuying Li, Jing Sun, Aixin Wang, Yequan Artificial Intelligence We introduce EgoMem, the first lifelong memory agent tailored for full-duplex models that process real-time omnimodal streams. EgoMem enables real-time models to recognize multiple users directly from raw audiovisual streams, to provide personalized response, and to maintain long-term knowledge of users' facts, preferences, and social relationships extracted from audiovisual history. EgoMem operates with three asynchronous processes: (i) a retrieval process that dynamically identifies user via face and voice, and gathers relevant context from a long-term memory; (ii) an omnimodal dialog process that generates personalized audio responses based on the retrieved context; and (iii) a memory management process that automatically detects dialog boundaries from omnimodal streams, and extracts necessary information to update the long-term memory. Unlike existing memory agents for LLMs, EgoMem relies entirely on raw audiovisual streams, making it especially suitable for lifelong, real-time, and embodied scenarios. Experimental results demonstrate that EgoMem's retrieval and memory management modules achieve over 95% accuracy on the test set. When integrated with a fine-tuned RoboEgo omnimodal chatbot, the system achieves fact-consistency scores above 87% in real-time personalized dialogs, establishing a strong baseline for future research. |
| title | EgoMem: Lifelong Memory Agent for Full-duplex Omnimodal Models |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.11914 |