UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910536365506560 |
|---|---|
| author | He, Chaoqun Luo, Renjie Hu, Shengding Zhao, Yuanqian Zhou, Jie Wu, Hanghao Zhang, Jiajie Han, Xu Liu, Zhiyuan Sun, Maosong |
| author_facet | He, Chaoqun Luo, Renjie Hu, Shengding Zhao, Yuanqian Zhou, Jie Wu, Hanghao Zhang, Jiajie Han, Xu Liu, Zhiyuan Sun, Maosong |
| contents | Evaluation is pivotal for refining Large Language Models (LLMs), pinpointing their capabilities, and guiding enhancements. The rapid development of LLMs calls for a lightweight and easy-to-use framework for swift evaluation deployment. However, considering various implementation details, developing a comprehensive evaluation platform is never easy. Existing platforms are often complex and poorly modularized, hindering seamless incorporation into research workflows. This paper introduces UltraEval, a user-friendly evaluation framework characterized by its lightweight nature, comprehensiveness, modularity, and efficiency. We identify and reimplement three core components of model evaluation (models, data, and metrics). The resulting composability allows for the free combination of different models, tasks, prompts, benchmarks, and metrics within a unified evaluation workflow. Additionally, UltraEval supports diverse models owing to a unified HTTP service and provides sufficient inference acceleration. UltraEval is now available for researchers publicly. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_07584 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs He, Chaoqun Luo, Renjie Hu, Shengding Zhao, Yuanqian Zhou, Jie Wu, Hanghao Zhang, Jiajie Han, Xu Liu, Zhiyuan Sun, Maosong Computation and Language Evaluation is pivotal for refining Large Language Models (LLMs), pinpointing their capabilities, and guiding enhancements. The rapid development of LLMs calls for a lightweight and easy-to-use framework for swift evaluation deployment. However, considering various implementation details, developing a comprehensive evaluation platform is never easy. Existing platforms are often complex and poorly modularized, hindering seamless incorporation into research workflows. This paper introduces UltraEval, a user-friendly evaluation framework characterized by its lightweight nature, comprehensiveness, modularity, and efficiency. We identify and reimplement three core components of model evaluation (models, data, and metrics). The resulting composability allows for the free combination of different models, tasks, prompts, benchmarks, and metrics within a unified evaluation workflow. Additionally, UltraEval supports diverse models owing to a unified HTTP service and provides sufficient inference acceleration. UltraEval is now available for researchers publicly. |
| title | UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.07584 |