Multi-Dimensional Evaluation of Text Summarization with In-Context Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910335553765376 |
|---|---|
| author | Jain, Sameer Keshava, Vaishakh Sathyendra, Swarnashree Mysore Fernandes, Patrick Liu, Pengfei Neubig, Graham Zhou, Chunting |
| author_facet | Jain, Sameer Keshava, Vaishakh Sathyendra, Swarnashree Mysore Fernandes, Patrick Liu, Pengfei Neubig, Graham Zhou, Chunting |
| contents | Evaluation of natural language generation (NLG) is complex and multi-dimensional. Generated text can be evaluated for fluency, coherence, factuality, or any other dimensions of interest. Most frameworks that perform such multi-dimensional evaluation require training on large manually or synthetically generated datasets. In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning, obviating the need for large training datasets. Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization, establishing state-of-the-art on dimensions such as relevance and factual consistency. We then analyze the effects of factors such as the selection and number of in-context examples on performance. Finally, we study the efficacy of in-context learning based evaluators in evaluating zero-shot summaries written by large language models such as GPT-3. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_01200 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Multi-Dimensional Evaluation of Text Summarization with In-Context Learning Jain, Sameer Keshava, Vaishakh Sathyendra, Swarnashree Mysore Fernandes, Patrick Liu, Pengfei Neubig, Graham Zhou, Chunting Computation and Language Evaluation of natural language generation (NLG) is complex and multi-dimensional. Generated text can be evaluated for fluency, coherence, factuality, or any other dimensions of interest. Most frameworks that perform such multi-dimensional evaluation require training on large manually or synthetically generated datasets. In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning, obviating the need for large training datasets. Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization, establishing state-of-the-art on dimensions such as relevance and factual consistency. We then analyze the effects of factors such as the selection and number of in-context examples on performance. Finally, we study the efficacy of in-context learning based evaluators in evaluating zero-shot summaries written by large language models such as GPT-3. |
| title | Multi-Dimensional Evaluation of Text Summarization with In-Context Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2306.01200 |