MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910655979716608 |
|---|---|
| author | Ni, Jinjie Song, Yifan Ghosal, Deepanway Li, Bo Zhang, David Junhao Yue, Xiang Xue, Fuzhao Zheng, Zian Zhang, Kaichen Shah, Mahir Jain, Kabir You, Yang Shieh, Michael |
| author_facet | Ni, Jinjie Song, Yifan Ghosal, Deepanway Li, Bo Zhang, David Junhao Yue, Xiang Xue, Fuzhao Zheng, Zian Zhang, Kaichen Shah, Mahir Jain, Kabir You, Yang Shieh, Michael |
| contents | Perceiving and generating diverse modalities are crucial for AI models to effectively learn from and engage with real-world signals, necessitating reliable evaluations for their development. We identify two major issues in current evaluations: (1) inconsistent standards, shaped by different communities with varying protocols and maturity levels; and (2) significant query, grading, and generalization biases. To address these, we introduce MixEval-X, the first any-to-any, real-world benchmark designed to optimize and standardize evaluations across diverse input and output modalities. We propose multi-modal benchmark mixture and adaptation-rectification pipelines to reconstruct real-world task distributions, ensuring evaluations generalize effectively to real-world use cases. Extensive meta-evaluations show our approach effectively aligns benchmark samples with real-world task distributions. Meanwhile, MixEval-X's model rankings correlate strongly with that of crowd-sourced real-world evaluations (up to 0.98) while being much more efficient. We provide comprehensive leaderboards to rerank existing models and organizations and offer insights to enhance understanding of multi-modal evaluations and inform future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_13754 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures Ni, Jinjie Song, Yifan Ghosal, Deepanway Li, Bo Zhang, David Junhao Yue, Xiang Xue, Fuzhao Zheng, Zian Zhang, Kaichen Shah, Mahir Jain, Kabir You, Yang Shieh, Michael Artificial Intelligence Machine Learning Multimedia Perceiving and generating diverse modalities are crucial for AI models to effectively learn from and engage with real-world signals, necessitating reliable evaluations for their development. We identify two major issues in current evaluations: (1) inconsistent standards, shaped by different communities with varying protocols and maturity levels; and (2) significant query, grading, and generalization biases. To address these, we introduce MixEval-X, the first any-to-any, real-world benchmark designed to optimize and standardize evaluations across diverse input and output modalities. We propose multi-modal benchmark mixture and adaptation-rectification pipelines to reconstruct real-world task distributions, ensuring evaluations generalize effectively to real-world use cases. Extensive meta-evaluations show our approach effectively aligns benchmark samples with real-world task distributions. Meanwhile, MixEval-X's model rankings correlate strongly with that of crowd-sourced real-world evaluations (up to 0.98) while being much more efficient. We provide comprehensive leaderboards to rerank existing models and organizations and offer insights to enhance understanding of multi-modal evaluations and inform future research. |
| title | MixEval-X: Any-to-Any Evaluations from Real-World Data Mixtures |
| topic | Artificial Intelligence Machine Learning Multimedia |
| url | https://arxiv.org/abs/2410.13754 |