Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.11591 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910645059846144 |
|---|---|
| author | Afzal, Anum Chalumattu, Ribin Matthes, Florian Mascarell, Laura |
| author_facet | Afzal, Anum Chalumattu, Ribin Matthes, Florian Mascarell, Laura |
| contents | Despite the advances in the abstractive summarization task using Large Language Models (LLM), there is a lack of research that asses their abilities to easily adapt to different domains. We evaluate the domain adaptation abilities of a wide range of LLMs on the summarization task across various domains in both fine-tuning and in-context learning settings. We also present AdaptEval, the first domain adaptation evaluation suite. AdaptEval includes a domain benchmark and a set of metrics to facilitate the analysis of domain adaptation. Our results demonstrate that LLMs exhibit comparable performance in the in-context learning setting, regardless of their parameter scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_11591 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization Afzal, Anum Chalumattu, Ribin Matthes, Florian Mascarell, Laura Computation and Language Despite the advances in the abstractive summarization task using Large Language Models (LLM), there is a lack of research that asses their abilities to easily adapt to different domains. We evaluate the domain adaptation abilities of a wide range of LLMs on the summarization task across various domains in both fine-tuning and in-context learning settings. We also present AdaptEval, the first domain adaptation evaluation suite. AdaptEval includes a domain benchmark and a set of metrics to facilitate the analysis of domain adaptation. Our results demonstrate that LLMs exhibit comparable performance in the in-context learning setting, regardless of their parameter scale. |
| title | AdaptEval: Evaluating Large Language Models on Domain Adaptation for Text Summarization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.11591 |