Memory Helps, but Confabulation Misleads: Understanding Streaming Events in Videos with MLLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909504010977280 |
|---|---|
| author | Zhang, Gengyuan Ding, Mingcong Liu, Tong Zhang, Yao Tresp, Volker |
| author_facet | Zhang, Gengyuan Ding, Mingcong Liu, Tong Zhang, Yao Tresp, Volker |
| contents | Multimodal large language models (MLLMs) have demonstrated strong performance in understanding videos holistically, yet their ability to process streaming videos-videos are treated as a sequence of visual events-remains underexplored. Intuitively, leveraging past events as memory can enrich contextual and temporal understanding of the current event. In this paper, we show that leveraging memories as contexts helps MLLMs better understand video events. However, because such memories rely on predictions of preceding events, they may contain misinformation, leading to confabulation and degraded performance. To address this, we propose a confabulation-aware memory modification method that mitigates confabulated memory for memory-enhanced event understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15457 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Memory Helps, but Confabulation Misleads: Understanding Streaming Events in Videos with MLLMs Zhang, Gengyuan Ding, Mingcong Liu, Tong Zhang, Yao Tresp, Volker Computer Vision and Pattern Recognition Multimodal large language models (MLLMs) have demonstrated strong performance in understanding videos holistically, yet their ability to process streaming videos-videos are treated as a sequence of visual events-remains underexplored. Intuitively, leveraging past events as memory can enrich contextual and temporal understanding of the current event. In this paper, we show that leveraging memories as contexts helps MLLMs better understand video events. However, because such memories rely on predictions of preceding events, they may contain misinformation, leading to confabulation and degraded performance. To address this, we propose a confabulation-aware memory modification method that mitigates confabulated memory for memory-enhanced event understanding. |
| title | Memory Helps, but Confabulation Misleads: Understanding Streaming Events in Videos with MLLMs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.15457 |