Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2503.12051 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916983888412672 |
|---|---|
| author | Gao, Fan Huang, Cheng Tashi, Nyima Wang, Xiangxiang Tsering, Thupten Ma-bao, Ban Duojie, Renzeg Luosang, Gadeng Dongrub, Rinchen Tashi, Dorje Feng, Hao Wang Xiao Yu, Yongbin |
| author_facet | Gao, Fan Huang, Cheng Tashi, Nyima Wang, Xiangxiang Tsering, Thupten Ma-bao, Ban Duojie, Renzeg Luosang, Gadeng Dongrub, Rinchen Tashi, Dorje Feng, Hao Wang Xiao Yu, Yongbin |
| contents | Large language models have made tremendous progress in recent years, but low-resource languages, like Tibetan, remain significantly underrepresented in their evaluation. Despite Tibetan being spoken by over seven million people, it has largely been neglected in the development and assessment of large language models. To address this gap, we present a \textbf{T}ibetan \textbf{L}anguage \textbf{U}nderstanding \textbf{E}valuation Benchmark, \textbf{TLUE}, the first large-scale benchmark for measuring the proficiency of LLMs in the Tibetan language. \textbf{TLUE} comprises two major components: a comprehensive multi-task understanding benchmark spanning 5 domains and 67 subdomains, and a safety benchmark encompassing 7 subdomains. Then, we evaluate a diverse set of state-of-the-art large language models. Experimental results demonstrate that most large language models perform below the random baseline, highlighting the considerable challenges they face in Tibetan language processing. \textbf{TLUE} provides a crucial foundation for advancing future research in Tibetan language understanding and highlights the importance of promoting greater inclusivity in the development of large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_12051 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TLUE: A Tibetan Language Understanding Evaluation Benchmark Gao, Fan Huang, Cheng Tashi, Nyima Wang, Xiangxiang Tsering, Thupten Ma-bao, Ban Duojie, Renzeg Luosang, Gadeng Dongrub, Rinchen Tashi, Dorje Feng, Hao Wang Xiao Yu, Yongbin Computation and Language Large language models have made tremendous progress in recent years, but low-resource languages, like Tibetan, remain significantly underrepresented in their evaluation. Despite Tibetan being spoken by over seven million people, it has largely been neglected in the development and assessment of large language models. To address this gap, we present a \textbf{T}ibetan \textbf{L}anguage \textbf{U}nderstanding \textbf{E}valuation Benchmark, \textbf{TLUE}, the first large-scale benchmark for measuring the proficiency of LLMs in the Tibetan language. \textbf{TLUE} comprises two major components: a comprehensive multi-task understanding benchmark spanning 5 domains and 67 subdomains, and a safety benchmark encompassing 7 subdomains. Then, we evaluate a diverse set of state-of-the-art large language models. Experimental results demonstrate that most large language models perform below the random baseline, highlighting the considerable challenges they face in Tibetan language processing. \textbf{TLUE} provides a crucial foundation for advancing future research in Tibetan language understanding and highlights the importance of promoting greater inclusivity in the development of large language models. |
| title | TLUE: A Tibetan Language Understanding Evaluation Benchmark |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.12051 |