Assessment of Multimodal Large Language Models in Alignment with Human Values
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911814593282048 |
|---|---|
| author | Shi, Zhelun Wang, Zhipin Fan, Hongxing Zhang, Zaibin Li, Lijun Zhang, Yongting Yin, Zhenfei Sheng, Lu Qiao, Yu Shao, Jing |
| author_facet | Shi, Zhelun Wang, Zhipin Fan, Hongxing Zhang, Zaibin Li, Lijun Zhang, Yongting Yin, Zhenfei Sheng, Lu Qiao, Yu Shao, Jing |
| contents | Large Language Models (LLMs) aim to serve as versatile assistants aligned with human values, as defined by the principles of being helpful, honest, and harmless (hhh). However, in terms of Multimodal Large Language Models (MLLMs), despite their commendable performance in perception and reasoning tasks, their alignment with human values remains largely unexplored, given the complexity of defining hhh dimensions in the visual world and the difficulty in collecting relevant data that accurately mirrors real-world situations. To address this gap, we introduce Ch3Ef, a Compreh3ensive Evaluation dataset and strategy for assessing alignment with human expectations. Ch3Ef dataset contains 1002 human-annotated data samples, covering 12 domains and 46 tasks based on the hhh principle. We also present a unified evaluation strategy supporting assessment across various scenarios and different perspectives. Based on the evaluation results, we summarize over 10 key findings that deepen the understanding of MLLM capabilities, limitations, and the dynamic relationships between evaluation levels, guiding future advancements in the field. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_17830 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Assessment of Multimodal Large Language Models in Alignment with Human Values Shi, Zhelun Wang, Zhipin Fan, Hongxing Zhang, Zaibin Li, Lijun Zhang, Yongting Yin, Zhenfei Sheng, Lu Qiao, Yu Shao, Jing Computer Vision and Pattern Recognition Large Language Models (LLMs) aim to serve as versatile assistants aligned with human values, as defined by the principles of being helpful, honest, and harmless (hhh). However, in terms of Multimodal Large Language Models (MLLMs), despite their commendable performance in perception and reasoning tasks, their alignment with human values remains largely unexplored, given the complexity of defining hhh dimensions in the visual world and the difficulty in collecting relevant data that accurately mirrors real-world situations. To address this gap, we introduce Ch3Ef, a Compreh3ensive Evaluation dataset and strategy for assessing alignment with human expectations. Ch3Ef dataset contains 1002 human-annotated data samples, covering 12 domains and 46 tasks based on the hhh principle. We also present a unified evaluation strategy supporting assessment across various scenarios and different perspectives. Based on the evaluation results, we summarize over 10 key findings that deepen the understanding of MLLM capabilities, limitations, and the dynamic relationships between evaluation levels, guiding future advancements in the field. |
| title | Assessment of Multimodal Large Language Models in Alignment with Human Values |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.17830 |