Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varieties
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916657701584896 |
|---|---|
| author | Tang, Zixin Huang, Chieh-Yang Li, Tsung-Che Ng, Ho Yin Sam Huang, Hen-Hsen Huang, Ting-Hao 'Kenneth' |
| author_facet | Tang, Zixin Huang, Chieh-Yang Li, Tsung-Che Ng, Ho Yin Sam Huang, Hen-Hsen Huang, Ting-Hao 'Kenneth' |
| contents | A language can have different varieties. These varieties can affect the performance of natural language processing (NLP) models, including large language models (LLMs), which are often trained on data from widely spoken varieties. This paper introduces a novel and cost-effective approach to benchmark model performance across language varieties. We argue that international online review platforms, such as Booking.com, can serve as effective data sources for constructing datasets that capture comments in different language varieties from similar real-world scenarios, like reviews for the same hotel with the same rating using the same language (e.g., Mandarin Chinese) but different language varieties (e.g., Taiwan Mandarin, Mainland Mandarin). To prove this concept, we constructed a contextually aligned dataset comprising reviews in Taiwan Mandarin and Mainland Mandarin and tested six LLMs in a sentiment analysis task. Our results show that LLMs consistently underperform in Taiwan Mandarin. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_07058 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varieties Tang, Zixin Huang, Chieh-Yang Li, Tsung-Che Ng, Ho Yin Sam Huang, Hen-Hsen Huang, Ting-Hao 'Kenneth' Computation and Language Human-Computer Interaction A language can have different varieties. These varieties can affect the performance of natural language processing (NLP) models, including large language models (LLMs), which are often trained on data from widely spoken varieties. This paper introduces a novel and cost-effective approach to benchmark model performance across language varieties. We argue that international online review platforms, such as Booking.com, can serve as effective data sources for constructing datasets that capture comments in different language varieties from similar real-world scenarios, like reviews for the same hotel with the same rating using the same language (e.g., Mandarin Chinese) but different language varieties (e.g., Taiwan Mandarin, Mainland Mandarin). To prove this concept, we constructed a contextually aligned dataset comprising reviews in Taiwan Mandarin and Mainland Mandarin and tested six LLMs in a sentiment analysis task. Our results show that LLMs consistently underperform in Taiwan Mandarin. |
| title | Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varieties |
| topic | Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2502.07058 |