Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913919671468032 |
|---|---|
| author | Montalan, Jann Railey Ngui, Jian Gang Leong, Wei Qi Susanto, Yosephine Rengarajan, Hamsawardhini Aji, Alham Fikri Tjhi, William Chandra |
| author_facet | Montalan, Jann Railey Ngui, Jian Gang Leong, Wei Qi Susanto, Yosephine Rengarajan, Hamsawardhini Aji, Alham Fikri Tjhi, William Chandra |
| contents | Multilingual large language models (LLMs) today may not necessarily provide culturally appropriate and relevant responses to its Filipino users. We introduce Kalahi, a cultural LLM evaluation suite collaboratively created by native Filipino speakers. It is composed of 150 high-quality, handcrafted and nuanced prompts that test LLMs for generations that are relevant to shared Filipino cultural knowledge and values. Strong LLM performance in Kalahi indicates a model's ability to generate responses similar to what an average Filipino would say or do in a given situation. We conducted experiments on LLMs with multilingual and Filipino language support. Results show that Kalahi, while trivial for Filipinos, is challenging for LLMs, with the best model answering only 46.0% of the questions correctly compared to native Filipino performance of 89.10%. Thus, Kalahi can be used to accurately and reliably evaluate Filipino cultural representation in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_15380 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino Montalan, Jann Railey Ngui, Jian Gang Leong, Wei Qi Susanto, Yosephine Rengarajan, Hamsawardhini Aji, Alham Fikri Tjhi, William Chandra Computation and Language Artificial Intelligence Multilingual large language models (LLMs) today may not necessarily provide culturally appropriate and relevant responses to its Filipino users. We introduce Kalahi, a cultural LLM evaluation suite collaboratively created by native Filipino speakers. It is composed of 150 high-quality, handcrafted and nuanced prompts that test LLMs for generations that are relevant to shared Filipino cultural knowledge and values. Strong LLM performance in Kalahi indicates a model's ability to generate responses similar to what an average Filipino would say or do in a given situation. We conducted experiments on LLMs with multilingual and Filipino language support. Results show that Kalahi, while trivial for Filipinos, is challenging for LLMs, with the best model answering only 46.0% of the questions correctly compared to native Filipino performance of 89.10%. Thus, Kalahi can be used to accurately and reliably evaluate Filipino cultural representation in LLMs. |
| title | Kalahi: A handcrafted, grassroots cultural LLM evaluation suite for Filipino |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2409.15380 |