Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2504.05898 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908306945081344 |
|---|---|
| author | Limkonchotiwat, Peerat Masuk, Kanruethai Nonesung, Surapon Mai-On, Chalermpun Nutanong, Sarana Ponwitayarat, Wuttikorn Manakul, Potsawee |
| author_facet | Limkonchotiwat, Peerat Masuk, Kanruethai Nonesung, Surapon Mai-On, Chalermpun Nutanong, Sarana Ponwitayarat, Wuttikorn Manakul, Potsawee |
| contents | Large language models show promising results in various NLP tasks. Despite these successes, the robustness and consistency of LLMs in underrepresented languages remain largely unexplored, especially concerning local dialects. Existing benchmarks also focus on main dialects, neglecting LLMs' ability on local dialect texts. In this paper, we introduce a Thai local dialect benchmark covering Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks. Furthermore, we propose a human evaluation guideline and metric for Thai local dialects to assess generation fluency and dialect-specific accuracy. Results show that LLM performance declines significantly in local Thai dialects compared to standard Thai, with only proprietary models like GPT-4o and Gemini2 demonstrating some fluency |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_05898 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation Limkonchotiwat, Peerat Masuk, Kanruethai Nonesung, Surapon Mai-On, Chalermpun Nutanong, Sarana Ponwitayarat, Wuttikorn Manakul, Potsawee Computation and Language Large language models show promising results in various NLP tasks. Despite these successes, the robustness and consistency of LLMs in underrepresented languages remain largely unexplored, especially concerning local dialects. Existing benchmarks also focus on main dialects, neglecting LLMs' ability on local dialect texts. In this paper, we introduce a Thai local dialect benchmark covering Northern (Lanna), Northeastern (Isan), and Southern (Dambro) Thai, evaluating LLMs on five NLP tasks: summarization, question answering, translation, conversation, and food-related tasks. Furthermore, we propose a human evaluation guideline and metric for Thai local dialects to assess generation fluency and dialect-specific accuracy. Results show that LLM performance declines significantly in local Thai dialects compared to standard Thai, with only proprietary models like GPT-4o and Gemini2 demonstrating some fluency |
| title | Assessing Thai Dialect Performance in LLMs with Automatic Benchmarks and Human Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.05898 |