Improving Domain Adaptation through Extended-Text Reading Comprehension
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911760114515968 |
|---|---|
| author | Jiang, Ting Huang, Shaohan Luo, Shengyue Zhang, Zihan Huang, Haizhen Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi Wang, Deqing Zhuang, Fuzhen |
| author_facet | Jiang, Ting Huang, Shaohan Luo, Shengyue Zhang, Zihan Huang, Haizhen Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi Wang, Deqing Zhuang, Fuzhen |
| contents | To enhance the domain-specific capabilities of large language models, continued pre-training on a domain-specific corpus is a prevalent method. Recent work demonstrates that adapting models using reading comprehension data formatted by regex-based patterns can significantly improve performance on domain-specific tasks. However, regex-based patterns are incapable of parsing raw corpora using domain-specific knowledge. Furthermore, the question and answer pairs are extracted directly from the corpus in predefined formats offers limited context. To address this limitation, we improve reading comprehension via LLM and clustering. LLM focuses on leveraging domain knowledge within the corpus to refine comprehension stage, while clustering supplies relevant knowledge by extending the context to enrich reading stage. Additionally, our method incorporates parameter-efficient fine-tuning to improve the efficiency of domain adaptation. In comparison to AdaptLLM, our method achieves an improvement exceeding 5% in domain-specific tasks. Our code will available at https://github.com/microsoft/LMOps. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_07284 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Domain Adaptation through Extended-Text Reading Comprehension Jiang, Ting Huang, Shaohan Luo, Shengyue Zhang, Zihan Huang, Haizhen Wei, Furu Deng, Weiwei Sun, Feng Zhang, Qi Wang, Deqing Zhuang, Fuzhen Computation and Language To enhance the domain-specific capabilities of large language models, continued pre-training on a domain-specific corpus is a prevalent method. Recent work demonstrates that adapting models using reading comprehension data formatted by regex-based patterns can significantly improve performance on domain-specific tasks. However, regex-based patterns are incapable of parsing raw corpora using domain-specific knowledge. Furthermore, the question and answer pairs are extracted directly from the corpus in predefined formats offers limited context. To address this limitation, we improve reading comprehension via LLM and clustering. LLM focuses on leveraging domain knowledge within the corpus to refine comprehension stage, while clustering supplies relevant knowledge by extending the context to enrich reading stage. Additionally, our method incorporates parameter-efficient fine-tuning to improve the efficiency of domain adaptation. In comparison to AdaptLLM, our method achieves an improvement exceeding 5% in domain-specific tasks. Our code will available at https://github.com/microsoft/LMOps. |
| title | Improving Domain Adaptation through Extended-Text Reading Comprehension |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2401.07284 |