RepEval: Effective Text Evaluation with LLM Representation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909367248355328 |
|---|---|
| author | Sheng, Shuqian Xu, Yi Zhang, Tianhang Shen, Zanwei Fu, Luoyi Ding, Jiaxin Zhou, Lei Gan, Xiaoying Wang, Xinbing Zhou, Chenghu |
| author_facet | Sheng, Shuqian Xu, Yi Zhang, Tianhang Shen, Zanwei Fu, Luoyi Ding, Jiaxin Zhou, Lei Gan, Xiaoying Wang, Xinbing Zhou, Chenghu |
| contents | The era of Large Language Models (LLMs) raises new demands for automatic evaluation metrics, which should be adaptable to various application scenarios while maintaining low cost and effectiveness. Traditional metrics for automatic text evaluation are often tailored to specific scenarios, while LLM-based evaluation metrics are costly, requiring fine-tuning or rely heavily on the generation capabilities of LLMs. Besides, previous LLM-based metrics ignore the fact that, within the space of LLM representations, there exist direction vectors that indicate the estimation of text quality. To this end, we introduce RepEval, a metric that leverages the projection of LLM representations for evaluation. Through simple prompt modifications, RepEval can easily transition to various tasks, requiring only minimal sample pairs for direction vector construction. Results on fourteen datasets across two evaluation tasks demonstrate the high effectiveness of our method, which exhibits a higher correlation with human judgments than previous methods, even in complex evaluation scenarios involving pair-wise selection under nuanced aspects. Our work underscores the richness of information regarding text quality embedded within LLM representations, offering insights for the development of new metrics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_19563 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RepEval: Effective Text Evaluation with LLM Representation Sheng, Shuqian Xu, Yi Zhang, Tianhang Shen, Zanwei Fu, Luoyi Ding, Jiaxin Zhou, Lei Gan, Xiaoying Wang, Xinbing Zhou, Chenghu Computation and Language The era of Large Language Models (LLMs) raises new demands for automatic evaluation metrics, which should be adaptable to various application scenarios while maintaining low cost and effectiveness. Traditional metrics for automatic text evaluation are often tailored to specific scenarios, while LLM-based evaluation metrics are costly, requiring fine-tuning or rely heavily on the generation capabilities of LLMs. Besides, previous LLM-based metrics ignore the fact that, within the space of LLM representations, there exist direction vectors that indicate the estimation of text quality. To this end, we introduce RepEval, a metric that leverages the projection of LLM representations for evaluation. Through simple prompt modifications, RepEval can easily transition to various tasks, requiring only minimal sample pairs for direction vector construction. Results on fourteen datasets across two evaluation tasks demonstrate the high effectiveness of our method, which exhibits a higher correlation with human judgments than previous methods, even in complex evaluation scenarios involving pair-wise selection under nuanced aspects. Our work underscores the richness of information regarding text quality embedded within LLM representations, offering insights for the development of new metrics. |
| title | RepEval: Effective Text Evaluation with LLM Representation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.19563 |