Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.21678 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918477700268032 |
|---|---|
| author | Bo, Weihao Zhang, Shan Sun, Yanpeng Wu, Jingjing Xie, Qunyi Tan, Xiao Chen, Kunbin He, Wei Li, Xiaofan Zhao, Na Wang, Jingdong Li, Zechao |
| author_facet | Bo, Weihao Zhang, Shan Sun, Yanpeng Wu, Jingjing Xie, Qunyi Tan, Xiao Chen, Kunbin He, Wei Li, Xiaofan Zhao, Na Wang, Jingdong Li, Zechao |
| contents | MLLMs exhibit strong reasoning on isolated queries, yet they operate de novo -- solving each problem independently and often repeating the same mistakes. Existing memory-augmented agents mainly store past trajectories for reuse. However, trajectory-based memory suffers from brevity bias, gradually losing essential domain knowledge. More critically, even in truly multimodal problem-solving settings, it records only a single-modality trace of past behavior, failing to preserve how visual attention and logical reasoning jointly contributed to the solution. This is fundamentally misaligned with human cognition: semantic memory is both multimodal and integrated, preserving visual and abstract knowledge through coordinated but distinct representational streams. We thus introduce ViLoMem, a dual-stream memory framework that constructs compact, schema-based memory. It separately encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences. Following a grow-and-refine principle, the system incrementally accumulates and updates multimodal semantic knowledge -- preserving stable, generalizable strategies while avoiding catastrophic forgetting. Across six multimodal benchmarks, ViLoMem consistently improves pass@1 accuracy and substantially reduces repeated visual and logical errors. Ablations confirm the necessity of dual-stream memory with explicit distraction-hallucination separation, demonstrating the value of error-aware multimodal memory for lifelong and cross-domain agentic learning. Our project page is available at https://weihao-bo.github.io/ViLoMeo-page. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_21678 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Agentic Learner with Grow-and-Refine Multimodal Semantic Memory Bo, Weihao Zhang, Shan Sun, Yanpeng Wu, Jingjing Xie, Qunyi Tan, Xiao Chen, Kunbin He, Wei Li, Xiaofan Zhao, Na Wang, Jingdong Li, Zechao Artificial Intelligence Machine Learning MLLMs exhibit strong reasoning on isolated queries, yet they operate de novo -- solving each problem independently and often repeating the same mistakes. Existing memory-augmented agents mainly store past trajectories for reuse. However, trajectory-based memory suffers from brevity bias, gradually losing essential domain knowledge. More critically, even in truly multimodal problem-solving settings, it records only a single-modality trace of past behavior, failing to preserve how visual attention and logical reasoning jointly contributed to the solution. This is fundamentally misaligned with human cognition: semantic memory is both multimodal and integrated, preserving visual and abstract knowledge through coordinated but distinct representational streams. We thus introduce ViLoMem, a dual-stream memory framework that constructs compact, schema-based memory. It separately encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences. Following a grow-and-refine principle, the system incrementally accumulates and updates multimodal semantic knowledge -- preserving stable, generalizable strategies while avoiding catastrophic forgetting. Across six multimodal benchmarks, ViLoMem consistently improves pass@1 accuracy and substantially reduces repeated visual and logical errors. Ablations confirm the necessity of dual-stream memory with explicit distraction-hallucination separation, demonstrating the value of error-aware multimodal memory for lifelong and cross-domain agentic learning. Our project page is available at https://weihao-bo.github.io/ViLoMeo-page. |
| title | Agentic Learner with Grow-and-Refine Multimodal Semantic Memory |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2511.21678 |