Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911953443618816 |
|---|---|
| author | Liu, Yixin Fabbri, Alexander R. Chen, Jiawen Zhao, Yilun Han, Simeng Joty, Shafiq Liu, Pengfei Radev, Dragomir Wu, Chien-Sheng Cohan, Arman |
| author_facet | Liu, Yixin Fabbri, Alexander R. Chen, Jiawen Zhao, Yilun Han, Simeng Joty, Shafiq Liu, Pengfei Radev, Dragomir Wu, Chien-Sheng Cohan, Arman |
| contents | While large language models (LLMs) can already achieve strong performance on standard generic summarization benchmarks, their performance on more complex summarization task settings is less studied. Therefore, we benchmark LLMs on instruction controllable text summarization, where the model input consists of both a source article and a natural language requirement for desired summary characteristics. To this end, we curate an evaluation-only dataset for this task setting and conduct human evaluations of five LLM-based systems to assess their instruction-following capabilities in controllable summarization. We then benchmark LLM-based automatic evaluation for this task with 4 different evaluation protocols and 11 LLMs, resulting in 40 evaluation methods. Our study reveals that instruction controllable text summarization remains a challenging task for LLMs, since (1) all LLMs evaluated still make factual and other types of errors in their summaries; (2) no LLM-based evaluation methods can achieve a strong alignment with human annotators when judging the quality of candidate summaries; (3) different LLMs show large performance gaps in summary generation and evaluation capabilities. We make our collected benchmark InstruSum publicly available to facilitate future research in this direction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09184 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization Liu, Yixin Fabbri, Alexander R. Chen, Jiawen Zhao, Yilun Han, Simeng Joty, Shafiq Liu, Pengfei Radev, Dragomir Wu, Chien-Sheng Cohan, Arman Computation and Language Machine Learning While large language models (LLMs) can already achieve strong performance on standard generic summarization benchmarks, their performance on more complex summarization task settings is less studied. Therefore, we benchmark LLMs on instruction controllable text summarization, where the model input consists of both a source article and a natural language requirement for desired summary characteristics. To this end, we curate an evaluation-only dataset for this task setting and conduct human evaluations of five LLM-based systems to assess their instruction-following capabilities in controllable summarization. We then benchmark LLM-based automatic evaluation for this task with 4 different evaluation protocols and 11 LLMs, resulting in 40 evaluation methods. Our study reveals that instruction controllable text summarization remains a challenging task for LLMs, since (1) all LLMs evaluated still make factual and other types of errors in their summaries; (2) no LLM-based evaluation methods can achieve a strong alignment with human annotators when judging the quality of candidate summaries; (3) different LLMs show large performance gaps in summary generation and evaluation capabilities. We make our collected benchmark InstruSum publicly available to facilitate future research in this direction. |
| title | Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarization |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2311.09184 |