TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929642538008576 |
|---|---|
| author | Barboule, Camille Huynh, Viet-Phi Bufort, Adrien Chabot, Yoan Damnati, Géraldine Lecorvé, Gwénolé |
| author_facet | Barboule, Camille Huynh, Viet-Phi Bufort, Adrien Chabot, Yoan Damnati, Géraldine Lecorvé, Gwénolé |
| contents | Despite outstanding processes in many tasks, Large Language Models (LLMs) still lack accuracy when dealing with highly technical domains. Especially, telecommunications (telco) is a particularly challenging domain due the large amount of lexical, semantic and conceptual peculiarities. Yet, this domain holds many valuable use cases, directly linked to industrial needs. Hence, this paper studies how LLMs can be adapted to the telco domain. It reports our effort to (i) collect a massive corpus of domain-specific data (800M tokens, 80K instructions), (ii) perform adaptation using various methodologies, and (iii) benchmark them against larger generalist models in downstream tasks that require extensive knowledge of telecommunications. Our experiments on Llama-2-7b show that domain-adapted models can challenge the large generalist models. They also suggest that adaptation can be restricted to a unique instruction-tuning step, dicarding the need for any fine-tuning on raw texts beforehand. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_15891 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain Barboule, Camille Huynh, Viet-Phi Bufort, Adrien Chabot, Yoan Damnati, Géraldine Lecorvé, Gwénolé Computation and Language Artificial Intelligence Despite outstanding processes in many tasks, Large Language Models (LLMs) still lack accuracy when dealing with highly technical domains. Especially, telecommunications (telco) is a particularly challenging domain due the large amount of lexical, semantic and conceptual peculiarities. Yet, this domain holds many valuable use cases, directly linked to industrial needs. Hence, this paper studies how LLMs can be adapted to the telco domain. It reports our effort to (i) collect a massive corpus of domain-specific data (800M tokens, 80K instructions), (ii) perform adaptation using various methodologies, and (iii) benchmark them against larger generalist models in downstream tasks that require extensive knowledge of telecommunications. Our experiments on Llama-2-7b show that domain-adapted models can challenge the large generalist models. They also suggest that adaptation can be restricted to a unique instruction-tuning step, dicarding the need for any fine-tuning on raw texts beforehand. |
| title | TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.15891 |