Neural Text Normalization for Luxembourgish using Real-Life Variation Data
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929628607676416 |
|---|---|
| author | Lutgen, Anne-Marie Plum, Alistair Purschke, Christoph Plank, Barbara |
| author_facet | Lutgen, Anne-Marie Plum, Alistair Purschke, Christoph Plank, Barbara |
| contents | Orthographic variation is very common in Luxembourgish texts due to the absence of a fully-fledged standard variety. Additionally, developing NLP tools for Luxembourgish is a difficult task given the lack of annotated and parallel data, which is exacerbated by ongoing standardization. In this paper, we propose the first sequence-to-sequence normalization models using the ByT5 and mT5 architectures with training data obtained from word-level real-life variation data. We perform a fine-grained, linguistically-motivated evaluation to test byte-based, word-based and pipeline-based models for their strengths and weaknesses in text normalization. We show that our sequence model using real-life variation data is an effective approach for tailor-made normalization in Luxembourgish. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_09383 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Neural Text Normalization for Luxembourgish using Real-Life Variation Data Lutgen, Anne-Marie Plum, Alistair Purschke, Christoph Plank, Barbara Computation and Language Orthographic variation is very common in Luxembourgish texts due to the absence of a fully-fledged standard variety. Additionally, developing NLP tools for Luxembourgish is a difficult task given the lack of annotated and parallel data, which is exacerbated by ongoing standardization. In this paper, we propose the first sequence-to-sequence normalization models using the ByT5 and mT5 architectures with training data obtained from word-level real-life variation data. We perform a fine-grained, linguistically-motivated evaluation to test byte-based, word-based and pipeline-based models for their strengths and weaknesses in text normalization. We show that our sequence model using real-life variation data is an effective approach for tailor-made normalization in Luxembourgish. |
| title | Neural Text Normalization for Luxembourgish using Real-Life Variation Data |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.09383 |