MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908783407529984 |
|---|---|
| author | Buaphet, Weerayut Nguyen, Thanh-Nhi Kondo, Risa Kajiwara, Tomoyuki Kim, Yumin Lee, Jimin Lee, Hwanhee Lovenia, Holy Limkonchotiwat, Peerat Nutanong, Sarana Van der Goot, Rob |
| author_facet | Buaphet, Weerayut Nguyen, Thanh-Nhi Kondo, Risa Kajiwara, Tomoyuki Kim, Yumin Lee, Jimin Lee, Hwanhee Lovenia, Holy Limkonchotiwat, Peerat Nutanong, Sarana Van der Goot, Rob |
| contents | Social media data has been of interest to Natural Language Processing (NLP) practitioners for over a decade, because of its richness in information, but also challenges for automatic processing. Since language use is more informal, spontaneous, and adheres to many different sociolects, the performance of NLP models often deteriorates. One solution to this problem is to transform data to a standard variant before processing it, which is also called lexical normalization. There has been a wide variety of benchmarks and models proposed for this task. The MultiLexNorm benchmark proposed to unify these efforts, but it consists almost solely of languages from the Indo-European language family in the Latin script. Hence, we propose an extension to MultiLexNorm, which covers 5 Asian languages from different language families in 4 different scripts. We show that the previous state-of-the-art model performs worse on the new languages and propose a new architecture based on Large Language Models (LLMs), which shows more robust performance. Finally, we analyze remaining errors, revealing future directions for this task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_16623 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages Buaphet, Weerayut Nguyen, Thanh-Nhi Kondo, Risa Kajiwara, Tomoyuki Kim, Yumin Lee, Jimin Lee, Hwanhee Lovenia, Holy Limkonchotiwat, Peerat Nutanong, Sarana Van der Goot, Rob Computation and Language Social media data has been of interest to Natural Language Processing (NLP) practitioners for over a decade, because of its richness in information, but also challenges for automatic processing. Since language use is more informal, spontaneous, and adheres to many different sociolects, the performance of NLP models often deteriorates. One solution to this problem is to transform data to a standard variant before processing it, which is also called lexical normalization. There has been a wide variety of benchmarks and models proposed for this task. The MultiLexNorm benchmark proposed to unify these efforts, but it consists almost solely of languages from the Indo-European language family in the Latin script. Hence, we propose an extension to MultiLexNorm, which covers 5 Asian languages from different language families in 4 different scripts. We show that the previous state-of-the-art model performs worse on the new languages and propose a new architecture based on Large Language Models (LLMs), which shows more robust performance. Finally, we analyze remaining errors, revealing future directions for this task. |
| title | MultiLexNorm++: A Unified Benchmark and a Generative Model for Lexical Normalization for Asian Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.16623 |