Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2402.12036 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913243641937920 |
|---|---|
| author | Belfathi, Anas Gallina, Ygor Hernandez, Nicolas Dufour, Richard Monceaux, Laura |
| author_facet | Belfathi, Anas Gallina, Ygor Hernandez, Nicolas Dufour, Richard Monceaux, Laura |
| contents | Recent advances in pre-trained language modeling have facilitated significant progress across various natural language processing (NLP) tasks. Word masking during model training constitutes a pivotal component of language modeling in architectures like BERT. However, the prevalent method of word masking relies on random selection, potentially disregarding domain-specific linguistic attributes. In this article, we introduce an innovative masking approach leveraging genre and topicality information to tailor language models to specialized domains. Our method incorporates a ranking process that prioritizes words based on their significance, subsequently guiding the masking procedure. Experiments conducted using continual pre-training within the legal domain have underscored the efficacy of our approach on the LegalGLUE benchmark in the English language. Pre-trained language models and code are freely available for use. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_12036 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristics Belfathi, Anas Gallina, Ygor Hernandez, Nicolas Dufour, Richard Monceaux, Laura Computation and Language Recent advances in pre-trained language modeling have facilitated significant progress across various natural language processing (NLP) tasks. Word masking during model training constitutes a pivotal component of language modeling in architectures like BERT. However, the prevalent method of word masking relies on random selection, potentially disregarding domain-specific linguistic attributes. In this article, we introduce an innovative masking approach leveraging genre and topicality information to tailor language models to specialized domains. Our method incorporates a ranking process that prioritizes words based on their significance, subsequently guiding the masking procedure. Experiments conducted using continual pre-training within the legal domain have underscored the efficacy of our approach on the LegalGLUE benchmark in the English language. Pre-trained language models and code are freely available for use. |
| title | Language Model Adaptation to Specialized Domains through Selective Masking based on Genre and Topical Characteristics |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.12036 |