Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.18218 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917813611921408 |
|---|---|
| author | Kroll, Margaret Kraus, Kelsey |
| author_facet | Kroll, Margaret Kraus, Kelsey |
| contents | The emergence of powerful LLMs has led to a paradigm shift in abstractive summarization of spoken documents. The properties that make LLMs so valuable for this task -- creativity, ability to produce fluent speech, and ability to abstract information from large corpora -- also present new challenges to evaluating their content. Quick, cost-effective automatic evaluations such as ROUGE and BERTScore offer promise, but do not yet show competitive performance when compared to human evaluations. We draw on methodologies from the social sciences to propose an evaluation paradigm for spoken document summarization explicitly tailored for generative AI content. We provide detailed evaluation criteria and best practices guidelines to ensure robustness in the experimental design, replicability, and trustworthiness of human evaluation studies. We additionally include two case studies that show how these human-in-the-loop evaluation methods have been implemented at a major U.S. technology company. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_18218 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Optimizing the role of human evaluation in LLM-based spoken document summarization systems Kroll, Margaret Kraus, Kelsey Artificial Intelligence Computation and Language Sound Audio and Speech Processing The emergence of powerful LLMs has led to a paradigm shift in abstractive summarization of spoken documents. The properties that make LLMs so valuable for this task -- creativity, ability to produce fluent speech, and ability to abstract information from large corpora -- also present new challenges to evaluating their content. Quick, cost-effective automatic evaluations such as ROUGE and BERTScore offer promise, but do not yet show competitive performance when compared to human evaluations. We draw on methodologies from the social sciences to propose an evaluation paradigm for spoken document summarization explicitly tailored for generative AI content. We provide detailed evaluation criteria and best practices guidelines to ensure robustness in the experimental design, replicability, and trustworthiness of human evaluation studies. We additionally include two case studies that show how these human-in-the-loop evaluation methods have been implemented at a major U.S. technology company. |
| title | Optimizing the role of human evaluation in LLM-based spoken document summarization systems |
| topic | Artificial Intelligence Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2410.18218 |