DACP: Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarization
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916999112687616 |
|---|---|
| author | Fu, Xue-Yong Khasanova, Elena Laskar, Md Tahmid Rahman Saini, Harsh TN, Shashi Bhushan |
| author_facet | Fu, Xue-Yong Khasanova, Elena Laskar, Md Tahmid Rahman Saini, Harsh TN, Shashi Bhushan |
| contents | Large language models (LLMs) have achieved impressive performance in text summarization, yet their performance often falls short when applied to specialized domains that differ from their original pre-training distribution. While fine-tuning can improve summarization quality, it typically relies on costly and scarce high-quality labeled data. In this work, we explore continual pre-training as a scalable, self-supervised approach to adapt LLMs for downstream summarization tasks, particularly in the context of noisy real-world conversation transcripts. We conduct extensive experiments using large-scale, unlabeled business conversation data to investigate whether continual pre-training enhances model capabilities in conversational summarization. Our results demonstrate that continual pre-training yields substantial gains in both in-domain and out-of-domain summarization benchmarks, while maintaining strong generalization and robustness. We also analyze the effects of data selection strategies, providing practical guidelines for applying continual pre-training in summarization-focused industrial applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_05858 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DACP: Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarization Fu, Xue-Yong Khasanova, Elena Laskar, Md Tahmid Rahman Saini, Harsh TN, Shashi Bhushan Computation and Language Artificial Intelligence Machine Learning Large language models (LLMs) have achieved impressive performance in text summarization, yet their performance often falls short when applied to specialized domains that differ from their original pre-training distribution. While fine-tuning can improve summarization quality, it typically relies on costly and scarce high-quality labeled data. In this work, we explore continual pre-training as a scalable, self-supervised approach to adapt LLMs for downstream summarization tasks, particularly in the context of noisy real-world conversation transcripts. We conduct extensive experiments using large-scale, unlabeled business conversation data to investigate whether continual pre-training enhances model capabilities in conversational summarization. Our results demonstrate that continual pre-training yields substantial gains in both in-domain and out-of-domain summarization benchmarks, while maintaining strong generalization and robustness. We also analyze the effects of data selection strategies, providing practical guidelines for applying continual pre-training in summarization-focused industrial applications. |
| title | DACP: Domain-Adaptive Continual Pre-Training of Large Language Models for Phone Conversation Summarization |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2510.05858 |