LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914690935816192 |
|---|---|
| author | Dalvi, Fahim Hasanain, Maram Boughorbel, Sabri Mousi, Basel Abdaljalil, Samir Nazar, Nizi Abdelali, Ahmed Chowdhury, Shammur Absar Mubarak, Hamdy Ali, Ahmed Hawasly, Majd Durrani, Nadir Alam, Firoj |
| author_facet | Dalvi, Fahim Hasanain, Maram Boughorbel, Sabri Mousi, Basel Abdaljalil, Samir Nazar, Nizi Abdelali, Ahmed Chowdhury, Shammur Absar Mubarak, Hamdy Ali, Ahmed Hawasly, Majd Durrani, Nadir Alam, Firoj |
| contents | The recent development and success of Large Language Models (LLMs) necessitate an evaluation of their performance across diverse NLP tasks in different languages. Although several frameworks have been developed and made publicly available, their customization capabilities for specific tasks and datasets are often complex for different users. In this study, we introduce the LLMeBench framework, which can be seamlessly customized to evaluate LLMs for any NLP task, regardless of language. The framework features generic dataset loaders, several model providers, and pre-implements most standard evaluation metrics. It supports in-context learning with zero- and few-shot settings. A specific dataset and task can be evaluated for a given LLM in less than 20 lines of code while allowing full flexibility to extend the framework for custom datasets, models, or tasks. The framework has been tested on 31 unique NLP tasks using 53 publicly available datasets within 90 experimental setups, involving approximately 296K data points. We open-sourced LLMeBench for the community (https://github.com/qcri/LLMeBench/) and a video demonstrating the framework is available online. (https://youtu.be/9cC2m_abk3A) |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_04945 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking Dalvi, Fahim Hasanain, Maram Boughorbel, Sabri Mousi, Basel Abdaljalil, Samir Nazar, Nizi Abdelali, Ahmed Chowdhury, Shammur Absar Mubarak, Hamdy Ali, Ahmed Hawasly, Majd Durrani, Nadir Alam, Firoj Computation and Language Artificial Intelligence 68T50 F.2.2; I.2.7 The recent development and success of Large Language Models (LLMs) necessitate an evaluation of their performance across diverse NLP tasks in different languages. Although several frameworks have been developed and made publicly available, their customization capabilities for specific tasks and datasets are often complex for different users. In this study, we introduce the LLMeBench framework, which can be seamlessly customized to evaluate LLMs for any NLP task, regardless of language. The framework features generic dataset loaders, several model providers, and pre-implements most standard evaluation metrics. It supports in-context learning with zero- and few-shot settings. A specific dataset and task can be evaluated for a given LLM in less than 20 lines of code while allowing full flexibility to extend the framework for custom datasets, models, or tasks. The framework has been tested on 31 unique NLP tasks using 53 publicly available datasets within 90 experimental setups, involving approximately 296K data points. We open-sourced LLMeBench for the community (https://github.com/qcri/LLMeBench/) and a video demonstrating the framework is available online. (https://youtu.be/9cC2m_abk3A) |
| title | LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking |
| topic | Computation and Language Artificial Intelligence 68T50 F.2.2; I.2.7 |
| url | https://arxiv.org/abs/2308.04945 |