Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.11903 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909531147075584 |
|---|---|
| author | Xue, Haochen Tang, Feilong Hu, Ming Liu, Yexin Huang, Qidong Li, Yulong Liu, Chengzhi Xu, Zhongxing Zhang, Chong Feng, Chun-Mei Xie, Yutong Razzak, Imran Ge, Zongyuan Su, Jionglong He, Junjun Qiao, Yu |
| author_facet | Xue, Haochen Tang, Feilong Hu, Ming Liu, Yexin Huang, Qidong Li, Yulong Liu, Chengzhi Xu, Zhongxing Zhang, Chong Feng, Chun-Mei Xie, Yutong Razzak, Imran Ge, Zongyuan Su, Jionglong He, Junjun Qiao, Yu |
| contents | Recent multimodal large language models (MLLMs) have demonstrated significant potential in open-ended conversation, generating more accurate and personalized responses. However, their abilities to memorize, recall, and reason in sustained interactions within real-world scenarios remain underexplored. This paper introduces MMRC, a Multi-Modal Real-world Conversation benchmark for evaluating six core open-ended abilities of MLLMs: information extraction, multi-turn reasoning, information update, image management, memory recall, and answer refusal. With data collected from real-world scenarios, MMRC comprises 5,120 conversations and 28,720 corresponding manually labeled questions, posing a significant challenge to existing MLLMs. Evaluations on 20 MLLMs in MMRC indicate an accuracy drop during open-ended interactions. We identify four common failure patterns: long-term memory degradation, inadequacies in updating factual knowledge, accumulated assumption of error propagation, and reluctance to say no. To mitigate these issues, we propose a simple yet effective NOTE-TAKING strategy, which can record key information from the conversation and remind the model during its responses, enhancing conversational capabilities. Experiments across six MLLMs demonstrate significant performance improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11903 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation Xue, Haochen Tang, Feilong Hu, Ming Liu, Yexin Huang, Qidong Li, Yulong Liu, Chengzhi Xu, Zhongxing Zhang, Chong Feng, Chun-Mei Xie, Yutong Razzak, Imran Ge, Zongyuan Su, Jionglong He, Junjun Qiao, Yu Computation and Language Recent multimodal large language models (MLLMs) have demonstrated significant potential in open-ended conversation, generating more accurate and personalized responses. However, their abilities to memorize, recall, and reason in sustained interactions within real-world scenarios remain underexplored. This paper introduces MMRC, a Multi-Modal Real-world Conversation benchmark for evaluating six core open-ended abilities of MLLMs: information extraction, multi-turn reasoning, information update, image management, memory recall, and answer refusal. With data collected from real-world scenarios, MMRC comprises 5,120 conversations and 28,720 corresponding manually labeled questions, posing a significant challenge to existing MLLMs. Evaluations on 20 MLLMs in MMRC indicate an accuracy drop during open-ended interactions. We identify four common failure patterns: long-term memory degradation, inadequacies in updating factual knowledge, accumulated assumption of error propagation, and reluctance to say no. To mitigate these issues, we propose a simple yet effective NOTE-TAKING strategy, which can record key information from the conversation and remind the model during its responses, enhancing conversational capabilities. Experiments across six MLLMs demonstrate significant performance improvements. |
| title | MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.11903 |