FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910408600715264 |
|---|---|
| author | Ye, Seonghyeon Kim, Doyoung Kim, Sungdong Hwang, Hyeonbin Kim, Seungone Jo, Yongrae Thorne, James Kim, Juho Seo, Minjoon |
| author_facet | Ye, Seonghyeon Kim, Doyoung Kim, Sungdong Hwang, Hyeonbin Kim, Seungone Jo, Yongrae Thorne, James Kim, Juho Seo, Minjoon |
| contents | Evaluation of Large Language Models (LLMs) is challenging because instruction-following necessitates alignment with human values and the required set of skills varies depending on the instruction. However, previous studies have mainly focused on coarse-grained evaluation (i.e. overall preference-based evaluation), which limits interpretability since it does not consider the nature of user instructions that require instance-wise skill composition. In this paper, we introduce FLASK (Fine-grained Language Model Evaluation based on Alignment Skill Sets), a fine-grained evaluation protocol for both human-based and model-based evaluation which decomposes coarse-level scoring to a skill set-level scoring for each instruction. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance and increasing the reliability of the evaluation. Using FLASK, we compare multiple open-source and proprietary LLMs and observe a high correlation between model-based and human-based evaluations. We publicly release the evaluation data and code implementation at https://github.com/kaistAI/FLASK. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_10928 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets Ye, Seonghyeon Kim, Doyoung Kim, Sungdong Hwang, Hyeonbin Kim, Seungone Jo, Yongrae Thorne, James Kim, Juho Seo, Minjoon Computation and Language Artificial Intelligence Evaluation of Large Language Models (LLMs) is challenging because instruction-following necessitates alignment with human values and the required set of skills varies depending on the instruction. However, previous studies have mainly focused on coarse-grained evaluation (i.e. overall preference-based evaluation), which limits interpretability since it does not consider the nature of user instructions that require instance-wise skill composition. In this paper, we introduce FLASK (Fine-grained Language Model Evaluation based on Alignment Skill Sets), a fine-grained evaluation protocol for both human-based and model-based evaluation which decomposes coarse-level scoring to a skill set-level scoring for each instruction. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance and increasing the reliability of the evaluation. Using FLASK, we compare multiple open-source and proprietary LLMs and observe a high correlation between model-based and human-based evaluations. We publicly release the evaluation data and code implementation at https://github.com/kaistAI/FLASK. |
| title | FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2307.10928 |