Enterprise Benchmarks for Large Language Model Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917805952073728 |
|---|---|
| author | Zhang, Bing Takeuchi, Mikio Kawahara, Ryo Asthana, Shubhi Hossain, Md. Maruf Ren, Guang-Jie Soule, Kate Zhu, Yada |
| author_facet | Zhang, Bing Takeuchi, Mikio Kawahara, Ryo Asthana, Shubhi Hossain, Md. Maruf Ren, Guang-Jie Soule, Kate Zhu, Yada |
| contents | The advancement of large language models (LLMs) has led to a greater challenge of having a rigorous and systematic evaluation of complex tasks performed, especially in enterprise applications. Therefore, LLMs need to be able to benchmark enterprise datasets for various tasks. This work presents a systematic exploration of benchmarking strategies tailored to LLM evaluation, focusing on the utilization of domain-specific datasets and consisting of a variety of NLP tasks. The proposed evaluation framework encompasses 25 publicly available datasets from diverse enterprise domains like financial services, legal, cyber security, and climate and sustainability. The diverse performance of 13 models across different enterprise tasks highlights the importance of selecting the right model based on the specific requirements of each task. Code and prompts are available on GitHub. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12857 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enterprise Benchmarks for Large Language Model Evaluation Zhang, Bing Takeuchi, Mikio Kawahara, Ryo Asthana, Shubhi Hossain, Md. Maruf Ren, Guang-Jie Soule, Kate Zhu, Yada Computation and Language Artificial Intelligence Computational Engineering, Finance, and Science The advancement of large language models (LLMs) has led to a greater challenge of having a rigorous and systematic evaluation of complex tasks performed, especially in enterprise applications. Therefore, LLMs need to be able to benchmark enterprise datasets for various tasks. This work presents a systematic exploration of benchmarking strategies tailored to LLM evaluation, focusing on the utilization of domain-specific datasets and consisting of a variety of NLP tasks. The proposed evaluation framework encompasses 25 publicly available datasets from diverse enterprise domains like financial services, legal, cyber security, and climate and sustainability. The diverse performance of 13 models across different enterprise tasks highlights the importance of selecting the right model based on the specific requirements of each task. Code and prompts are available on GitHub. |
| title | Enterprise Benchmarks for Large Language Model Evaluation |
| topic | Computation and Language Artificial Intelligence Computational Engineering, Finance, and Science |
| url | https://arxiv.org/abs/2410.12857 |