Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910617488588800 |
|---|---|
| author | Christophe, Clément Raha, Tathagata Maslenkova, Svetlana Salman, Muhammad Umar Kanithi, Praveen K Pimentel, Marco AF Khan, Shadab |
| author_facet | Christophe, Clément Raha, Tathagata Maslenkova, Svetlana Salman, Muhammad Umar Kanithi, Praveen K Pimentel, Marco AF Khan, Shadab |
| contents | Large Language Models (LLMs) have demonstrated significant potential in transforming clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning dataset of 500 million tokens. Our evaluation across various clinical tasks reveals the impact of each technique. While continuous pretraining beyond 250 billion tokens yields marginal improvements on its own, it establishes a strong foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on our benchmark. Complex prompt engineering methods further enhance performance. These findings show the importance of tailoring fine-tuning strategies and exploring innovative techniques to optimize LLM performance in the clinical domain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_14988 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs Christophe, Clément Raha, Tathagata Maslenkova, Svetlana Salman, Muhammad Umar Kanithi, Praveen K Pimentel, Marco AF Khan, Shadab Computation and Language Large Language Models (LLMs) have demonstrated significant potential in transforming clinical applications. In this study, we investigate the efficacy of four techniques in adapting LLMs for clinical use-cases: continuous pretraining, instruct fine-tuning, NEFTune, and prompt engineering. We employ these methods on Mistral 7B and Mixtral 8x7B models, leveraging a large-scale clinical pretraining dataset of 50 billion tokens and an instruct fine-tuning dataset of 500 million tokens. Our evaluation across various clinical tasks reveals the impact of each technique. While continuous pretraining beyond 250 billion tokens yields marginal improvements on its own, it establishes a strong foundation for instruct fine-tuning. Notably, NEFTune, designed primarily to enhance generation quality, surprisingly demonstrates additional gains on our benchmark. Complex prompt engineering methods further enhance performance. These findings show the importance of tailoring fine-tuning strategies and exploring innovative techniques to optimize LLM performance in the clinical domain. |
| title | Beyond Fine-tuning: Unleashing the Potential of Continuous Pretraining for Clinical LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.14988 |