Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.06011 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909246542577664 |
|---|---|
| author | Buonocore, Tommaso Mario Rancati, Simone Parimbelli, Enea |
| author_facet | Buonocore, Tommaso Mario Rancati, Simone Parimbelli, Enea |
| contents | The development of domain-specific language models has significantly advanced natural language processing applications in various specialized fields, particularly in biomedicine. However, the focus has largely been on English-language models, leaving a gap for less-resourced languages such as Italian. This paper introduces Igea, the first decoder-only language model designed explicitly for biomedical text generation in Italian. Built on the Minerva model and continually pretrained on a diverse corpus of Italian medical texts, Igea is available in three model sizes: 350 million, 1 billion, and 3 billion parameters. The models aim to balance computational efficiency and performance, addressing the challenges of managing the peculiarities of medical terminology in Italian. We evaluate Igea using a mix of in-domain biomedical corpora and general-purpose benchmarks, highlighting its efficacy and retention of general knowledge even after the domain-specific training. This paper discusses the model's development and evaluation, providing a foundation for future advancements in Italian biomedical NLP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_06011 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Igea: a Decoder-Only Language Model for Biomedical Text Generation in Italian Buonocore, Tommaso Mario Rancati, Simone Parimbelli, Enea Computation and Language Artificial Intelligence I.2.7; J.3 The development of domain-specific language models has significantly advanced natural language processing applications in various specialized fields, particularly in biomedicine. However, the focus has largely been on English-language models, leaving a gap for less-resourced languages such as Italian. This paper introduces Igea, the first decoder-only language model designed explicitly for biomedical text generation in Italian. Built on the Minerva model and continually pretrained on a diverse corpus of Italian medical texts, Igea is available in three model sizes: 350 million, 1 billion, and 3 billion parameters. The models aim to balance computational efficiency and performance, addressing the challenges of managing the peculiarities of medical terminology in Italian. We evaluate Igea using a mix of in-domain biomedical corpora and general-purpose benchmarks, highlighting its efficacy and retention of general knowledge even after the domain-specific training. This paper discusses the model's development and evaluation, providing a foundation for future advancements in Italian biomedical NLP. |
| title | Igea: a Decoder-Only Language Model for Biomedical Text Generation in Italian |
| topic | Computation and Language Artificial Intelligence I.2.7; J.3 |
| url | https://arxiv.org/abs/2407.06011 |