BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915339825053696 |
|---|---|
| author | Sounack, Thomas Davis, Joshua Durieux, Brigitte Chaffin, Antoine Pollard, Tom J. Lehman, Eric Johnson, Alistair E. W. McDermott, Matthew Naumann, Tristan Lindvall, Charlotta |
| author_facet | Sounack, Thomas Davis, Joshua Durieux, Brigitte Chaffin, Antoine Pollard, Tom J. Lehman, Eric Johnson, Alistair E. W. McDermott, Matthew Naumann, Tristan Lindvall, Charlotta |
| contents | Encoder-based transformer models are central to biomedical and clinical Natural Language Processing (NLP), as their bidirectional self-attention makes them well-suited for efficiently extracting structured information from unstructured text through discriminative tasks. However, encoders have seen slower development compared to decoder models, leading to limited domain adaptation in biomedical and clinical settings. We introduce BioClinical ModernBERT, a domain-adapted encoder that builds on the recent ModernBERT release, incorporating long-context processing and substantial improvements in speed and performance for biomedical and clinical NLP. BioClinical ModernBERT is developed through continued pretraining on the largest biomedical and clinical corpus to date, with over 53.5 billion tokens, and addresses a key limitation of prior clinical encoders by leveraging 20 datasets from diverse institutions, domains, and geographic regions, rather than relying on data from a single source. It outperforms existing biomedical and clinical encoders on four downstream tasks spanning a broad range of use cases. We release both base (150M parameters) and large (396M parameters) versions of BioClinical ModernBERT, along with training checkpoints to support further research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_10896 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP Sounack, Thomas Davis, Joshua Durieux, Brigitte Chaffin, Antoine Pollard, Tom J. Lehman, Eric Johnson, Alistair E. W. McDermott, Matthew Naumann, Tristan Lindvall, Charlotta Computation and Language Artificial Intelligence Encoder-based transformer models are central to biomedical and clinical Natural Language Processing (NLP), as their bidirectional self-attention makes them well-suited for efficiently extracting structured information from unstructured text through discriminative tasks. However, encoders have seen slower development compared to decoder models, leading to limited domain adaptation in biomedical and clinical settings. We introduce BioClinical ModernBERT, a domain-adapted encoder that builds on the recent ModernBERT release, incorporating long-context processing and substantial improvements in speed and performance for biomedical and clinical NLP. BioClinical ModernBERT is developed through continued pretraining on the largest biomedical and clinical corpus to date, with over 53.5 billion tokens, and addresses a key limitation of prior clinical encoders by leveraging 20 datasets from diverse institutions, domains, and geographic regions, rather than relying on data from a single source. It outperforms existing biomedical and clinical encoders on four downstream tasks spanning a broad range of use cases. We release both base (150M parameters) and large (396M parameters) versions of BioClinical ModernBERT, along with training checkpoints to support further research. |
| title | BioClinical ModernBERT: A State-of-the-Art Long-Context Encoder for Biomedical and Clinical NLP |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.10896 |