Can MLLMs Understand the Deep Implication Behind Chinese Images?
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913551559426048 |
|---|---|
| author | Zhang, Chenhao Feng, Xi Bai, Yuelin Du, Xinrun Hou, Jinchang Deng, Kaixin Han, Guangzeng Li, Qinrui Wang, Bingli Liu, Jiaheng Qu, Xingwei Zhang, Yifei Zhao, Qixuan Liang, Yiming Liu, Ziqiang Fang, Feiteng Yang, Min Huang, Wenhao Lin, Chenghua Zhang, Ge Ni, Shiwen |
| author_facet | Zhang, Chenhao Feng, Xi Bai, Yuelin Du, Xinrun Hou, Jinchang Deng, Kaixin Han, Guangzeng Li, Qinrui Wang, Bingli Liu, Jiaheng Qu, Xingwei Zhang, Yifei Zhao, Qixuan Liang, Yiming Liu, Ziqiang Fang, Feiteng Yang, Min Huang, Wenhao Lin, Chenghua Zhang, Ge Ni, Shiwen |
| contents | As the capabilities of Multimodal Large Language Models (MLLMs) continue to improve, the need for higher-order capability evaluation of MLLMs is increasing. However, there is a lack of work evaluating MLLM for higher-order perception and understanding of Chinese visual content. To fill the gap, we introduce the **C**hinese **I**mage **I**mplication understanding **Bench**mark, **CII-Bench**, which aims to assess the higher-order perception and understanding capabilities of MLLMs for Chinese images. CII-Bench stands out in several ways compared to existing benchmarks. Firstly, to ensure the authenticity of the Chinese context, images in CII-Bench are sourced from the Chinese Internet and manually reviewed, with corresponding answers also manually crafted. Additionally, CII-Bench incorporates images that represent Chinese traditional culture, such as famous Chinese traditional paintings, which can deeply reflect the model's understanding of Chinese traditional culture. Through extensive experiments on CII-Bench across multiple MLLMs, we have made significant findings. Initially, a substantial gap is observed between the performance of MLLMs and humans on CII-Bench. The highest accuracy of MLLMs attains 64.4%, where as human accuracy averages 78.2%, peaking at an impressive 81.0%. Subsequently, MLLMs perform worse on Chinese traditional culture images, suggesting limitations in their ability to understand high-level semantics and lack a deep knowledge base of Chinese traditional culture. Finally, it is observed that most models exhibit enhanced accuracy when image emotion hints are incorporated into the prompts. We believe that CII-Bench will enable MLLMs to gain a better understanding of Chinese semantics and Chinese-specific images, advancing the journey towards expert artificial general intelligence (AGI). Our project is publicly available at https://cii-bench.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_13854 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can MLLMs Understand the Deep Implication Behind Chinese Images? Zhang, Chenhao Feng, Xi Bai, Yuelin Du, Xinrun Hou, Jinchang Deng, Kaixin Han, Guangzeng Li, Qinrui Wang, Bingli Liu, Jiaheng Qu, Xingwei Zhang, Yifei Zhao, Qixuan Liang, Yiming Liu, Ziqiang Fang, Feiteng Yang, Min Huang, Wenhao Lin, Chenghua Zhang, Ge Ni, Shiwen Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Computers and Society As the capabilities of Multimodal Large Language Models (MLLMs) continue to improve, the need for higher-order capability evaluation of MLLMs is increasing. However, there is a lack of work evaluating MLLM for higher-order perception and understanding of Chinese visual content. To fill the gap, we introduce the **C**hinese **I**mage **I**mplication understanding **Bench**mark, **CII-Bench**, which aims to assess the higher-order perception and understanding capabilities of MLLMs for Chinese images. CII-Bench stands out in several ways compared to existing benchmarks. Firstly, to ensure the authenticity of the Chinese context, images in CII-Bench are sourced from the Chinese Internet and manually reviewed, with corresponding answers also manually crafted. Additionally, CII-Bench incorporates images that represent Chinese traditional culture, such as famous Chinese traditional paintings, which can deeply reflect the model's understanding of Chinese traditional culture. Through extensive experiments on CII-Bench across multiple MLLMs, we have made significant findings. Initially, a substantial gap is observed between the performance of MLLMs and humans on CII-Bench. The highest accuracy of MLLMs attains 64.4%, where as human accuracy averages 78.2%, peaking at an impressive 81.0%. Subsequently, MLLMs perform worse on Chinese traditional culture images, suggesting limitations in their ability to understand high-level semantics and lack a deep knowledge base of Chinese traditional culture. Finally, it is observed that most models exhibit enhanced accuracy when image emotion hints are incorporated into the prompts. We believe that CII-Bench will enable MLLMs to gain a better understanding of Chinese semantics and Chinese-specific images, advancing the journey towards expert artificial general intelligence (AGI). Our project is publicly available at https://cii-bench.github.io/. |
| title | Can MLLMs Understand the Deep Implication Behind Chinese Images? |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Computers and Society |
| url | https://arxiv.org/abs/2410.13854 |