Cover Image

FlagEvalMM: A Flexible Framework for Comprehensive Multimodal Model Evaluation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	He, Zheqi, Liu, Yesheng, Zheng, Jing-shu, Li, Xuejing, Yao, Jin-Ge, Qin, Bowen, Xuan, Richeng, Yang, Xi
Format:	Preprint
Published:	2025
Subjects:	Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

FlagEval Findings Report: A Preliminary Evaluation of Large Reasoning Models on Automatically Verifiable Textual and Visual Questions
by: Qin, Bowen, et al.
Published: (2025)

Beyond Multiple Choice: Verifiable OpenQA for Robust Vision-Language RFT
by: Liu, Yesheng, et al.
Published: (2025)

UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs
by: He, Chaoqun, et al.
Published: (2024)

LaoBench: A Large-Scale Multidimensional Lao Benchmark for Large Language Models
by: Gao, Jian, et al.
Published: (2025)

MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents
by: Li, Shilong, et al.
Published: (2025)

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning
by: He, Zheqi, et al.
Published: (2024)

MM-Eval: A Hierarchical Benchmark for Modern Mongolian Evaluation in LLMs
by: Zhang, Mengyuan, et al.
Published: (2024)

Eval4Sim: An Evaluation Framework for Persona Simulation
by: Bao, Eliseo, et al.
Published: (2026)

Bi-DCSpell: A Bi-directional Detector-Corrector Interactive Framework for Chinese Spelling Check
by: Wu, Haiming, et al.
Published: (2024)

MM-CRITIC: A Holistic Evaluation of Large Multimodal Models as Multimodal Critique
by: Zeng, Gailun, et al.
Published: (2025)

Searching from Area to Point: A Hierarchical Framework for Semantic-Geometric Combined Feature Matching
by: Zhang, Yesheng, et al.
Published: (2023)

MM-Eval: A Multilingual Meta-Evaluation Benchmark for LLM-as-a-Judge and Reward Models
by: Son, Guijin, et al.
Published: (2024)

HintEval: A Comprehensive Framework for Hint Generation and Evaluation for Questions
by: Mozafari, Jamshid, et al.
Published: (2025)

HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation
by: Luo, Wen, et al.
Published: (2024)

Do Vision-Language Models Measure Up? Benchmarking Visual Measurement Reading with MeasureBench
by: Lin, Fenfen, et al.
Published: (2025)

DeepResearchEval: An Automated Framework for Deep Research Task Construction and Agentic Evaluation
by: Wang, Yibo, et al.
Published: (2026)

QuantEval: A Benchmark for Financial Quantitative Tasks in Large Language Models
by: Kang, Zhaolu, et al.
Published: (2026)

Evaluating Attribute Comprehension in Large Vision-Language Models
by: Zhang, Haiwen, et al.
Published: (2024)

EmbodiedEval: Evaluate Multimodal LLMs as Embodied Agents
by: Cheng, Zhili, et al.
Published: (2025)

MM-Soc: Benchmarking Multimodal Large Language Models in Social Media Platforms
by: Jin, Yiqiao, et al.
Published: (2024)

MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities
by: Yu, Weihao, et al.
Published: (2023)

LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models
by: Zhang, Kaichen, et al.
Published: (2024)

AlphaEval: A Comprehensive and Efficient Evaluation Framework for Formula Alpha Mining
by: Ding, Hongjun, et al.
Published: (2025)

VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Model
by: Li, Xinhao, et al.
Published: (2024)

MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarios
by: Ruan, Jiacheng, et al.
Published: (2024)

CC-OCR: A Comprehensive and Challenging OCR Benchmark for Evaluating Large Multimodal Models in Literacy
by: Yang, Zhibo, et al.
Published: (2024)

MM-SAP: A Comprehensive Benchmark for Assessing Self-Awareness of Multimodal Large Language Models in Perception
by: Wang, Yuhao, et al.
Published: (2024)

MM-Snowball: Evaluating and Mitigating Hallucination Snowballing in Multimodal Multi-Turn Dialogue
by: Jiang, Yue, et al.
Published: (2026)

EvalCards: A Framework for Standardized Evaluation Reporting
by: Dhar, Ruchira, et al.
Published: (2025)

Cascaded Self-Evaluation Augmented Training for Lightweight Multimodal LLMs
by: Lv, Zheqi, et al.
Published: (2025)

MM-MATH: Advancing Multimodal Math Evaluation with Process Evaluation and Fine-grained Classification
by: Sun, Kai, et al.
Published: (2024)

DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection
by: Zhao, Kangran, et al.
Published: (2025)

GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation
by: Zong, Yi, et al.
Published: (2024)

MIPS at SemEval-2024 Task 3: Multimodal Emotion-Cause Pair Extraction in Conversations with Multimodal Language Models
by: Cheng, Zebang, et al.
Published: (2024)

Video-SafetyBench: A Benchmark for Safety Evaluation of Video LVLMs
by: Liu, Xuannan, et al.
Published: (2025)

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment
by: Zhang, Yi-Fan, et al.
Published: (2025)

MESA: Matching Everything by Segmenting Anything
by: Zhang, Yesheng, et al.
Published: (2024)

VF-Eval: Evaluating Multimodal LLMs for Generating Feedback on AIGC Videos
by: Song, Tingyu, et al.
Published: (2025)

TP-Eval: Tap Multimodal LLMs' Potential in Evaluation by Customizing Prompts
by: Xie, Yuxuan, et al.
Published: (2024)

SciSafeEval: A Comprehensive Benchmark for Safety Alignment of Large Language Models in Scientific Tasks
by: Li, Tianhao, et al.
Published: (2024)