Self-Evaluation of Large Language Model based on Glass-box Features
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913520554082304 |
|---|---|
| author | Huang, Hui Qu, Yingqi Liu, Jing Yang, Muyun Xu, Bing Zhao, Tiejun Lu, Wenpeng |
| author_facet | Huang, Hui Qu, Yingqi Liu, Jing Yang, Muyun Xu, Bing Zhao, Tiejun Lu, Wenpeng |
| contents | The proliferation of open-source Large Language Models (LLMs) underscores the pressing need for evaluation methods. Existing works primarily rely on external evaluators, focusing on training and prompting strategies. However, a crucial aspect, model-aware glass-box features, is overlooked. In this study, we explore the utility of glass-box features under the scenario of self-evaluation, namely applying an LLM to evaluate its own output. We investigate various glass-box feature groups and discovered that the softmax distribution serves as a reliable quality indicator for self-evaluation. Experimental results on public benchmarks validate the feasibility of self-evaluation of LLMs using glass-box features. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_04222 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Self-Evaluation of Large Language Model based on Glass-box Features Huang, Hui Qu, Yingqi Liu, Jing Yang, Muyun Xu, Bing Zhao, Tiejun Lu, Wenpeng Computation and Language The proliferation of open-source Large Language Models (LLMs) underscores the pressing need for evaluation methods. Existing works primarily rely on external evaluators, focusing on training and prompting strategies. However, a crucial aspect, model-aware glass-box features, is overlooked. In this study, we explore the utility of glass-box features under the scenario of self-evaluation, namely applying an LLM to evaluate its own output. We investigate various glass-box feature groups and discovered that the softmax distribution serves as a reliable quality indicator for self-evaluation. Experimental results on public benchmarks validate the feasibility of self-evaluation of LLMs using glass-box features. |
| title | Self-Evaluation of Large Language Model based on Glass-box Features |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.04222 |