Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.13663 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913618061164544 |
|---|---|
| author | Warner, Benjamin Chaffin, Antoine Clavié, Benjamin Weller, Orion Hallström, Oskar Taghadouini, Said Gallagher, Alexis Biswas, Raja Ladhak, Faisal Aarsen, Tom Cooper, Nathan Adams, Griffin Howard, Jeremy Poli, Iacopo |
| author_facet | Warner, Benjamin Chaffin, Antoine Clavié, Benjamin Weller, Orion Hallström, Oskar Taghadouini, Said Gallagher, Alexis Biswas, Raja Ladhak, Faisal Aarsen, Tom Cooper, Nathan Adams, Griffin Howard, Jeremy Poli, Iacopo |
| contents | Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13663 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference Warner, Benjamin Chaffin, Antoine Clavié, Benjamin Weller, Orion Hallström, Oskar Taghadouini, Said Gallagher, Alexis Biswas, Raja Ladhak, Faisal Aarsen, Tom Cooper, Nathan Adams, Griffin Howard, Jeremy Poli, Iacopo Computation and Language Artificial Intelligence Encoder-only transformer models such as BERT offer a great performance-size tradeoff for retrieval and classification tasks with respect to larger decoder-only models. Despite being the workhorse of numerous production pipelines, there have been limited Pareto improvements to BERT since its release. In this paper, we introduce ModernBERT, bringing modern model optimizations to encoder-only models and representing a major Pareto improvement over older encoders. Trained on 2 trillion tokens with a native 8192 sequence length, ModernBERT models exhibit state-of-the-art results on a large pool of evaluations encompassing diverse classification tasks and both single and multi-vector retrieval on different domains (including code). In addition to strong downstream performance, ModernBERT is also the most speed and memory efficient encoder and is designed for inference on common GPUs. |
| title | Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.13663 |