COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916216106385408 |
|---|---|
| author | Wibowo, Haryo Akbarianto Fuadi, Erland Hilman Nityasya, Made Nindyatama Prasojo, Radityo Eko Aji, Alham Fikri |
| author_facet | Wibowo, Haryo Akbarianto Fuadi, Erland Hilman Nityasya, Made Nindyatama Prasojo, Radityo Eko Aji, Alham Fikri |
| contents | We present COPAL-ID, a novel, public Indonesian language common sense reasoning dataset. Unlike the previous Indonesian COPA dataset (XCOPA-ID), COPAL-ID incorporates Indonesian local and cultural nuances, and therefore, provides a more natural portrayal of day-to-day causal reasoning within the Indonesian cultural sphere. Professionally written by natives from scratch, COPAL-ID is more fluent and free from awkward phrases, unlike the translated XCOPA-ID. In addition, we present COPAL-ID in both standard Indonesian and in Jakartan Indonesian-a dialect commonly used in daily conversation. COPAL-ID poses a greater challenge for existing open-sourced and closed state-of-the-art multilingual language models, yet is trivially easy for humans. Our findings suggest that general multilingual models struggle to perform well, achieving 66.91% accuracy on COPAL-ID. South-East Asian-specific models achieve slightly better performance of 73.88% accuracy. Yet, this number still falls short of near-perfect human performance. This shows that these language models are still way behind in comprehending the local nuances of Indonesian. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_01012 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances Wibowo, Haryo Akbarianto Fuadi, Erland Hilman Nityasya, Made Nindyatama Prasojo, Radityo Eko Aji, Alham Fikri Computation and Language 68T50 We present COPAL-ID, a novel, public Indonesian language common sense reasoning dataset. Unlike the previous Indonesian COPA dataset (XCOPA-ID), COPAL-ID incorporates Indonesian local and cultural nuances, and therefore, provides a more natural portrayal of day-to-day causal reasoning within the Indonesian cultural sphere. Professionally written by natives from scratch, COPAL-ID is more fluent and free from awkward phrases, unlike the translated XCOPA-ID. In addition, we present COPAL-ID in both standard Indonesian and in Jakartan Indonesian-a dialect commonly used in daily conversation. COPAL-ID poses a greater challenge for existing open-sourced and closed state-of-the-art multilingual language models, yet is trivially easy for humans. Our findings suggest that general multilingual models struggle to perform well, achieving 66.91% accuracy on COPAL-ID. South-East Asian-specific models achieve slightly better performance of 73.88% accuracy. Yet, this number still falls short of near-perfect human performance. This shows that these language models are still way behind in comprehending the local nuances of Indonesian. |
| title | COPAL-ID: Indonesian Language Reasoning with Local Culture and Nuances |
| topic | Computation and Language 68T50 |
| url | https://arxiv.org/abs/2311.01012 |