ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914426921156608 |
|---|---|
| author | Vieira, Inês Calvo, Inês Paulo, Iago Furtado, James Ferreira, Rafael Tavares, Diogo Glória-Silva, Diogo Semedo, David Magalhães, João |
| author_facet | Vieira, Inês Calvo, Inês Paulo, Iago Furtado, James Ferreira, Rafael Tavares, Diogo Glória-Silva, Diogo Semedo, David Magalhães, João |
| contents | As Large Language Models (LLMs) expand across multilingual domains, evaluating their performance in under-represented languages becomes increasingly important. European Portuguese (pt-PT) is particularly affected, as existing training data and benchmarks are mainly in Brazilian Portuguese (pt-BR). To address this, we introduce ALBA, a linguistically grounded benchmark designed from the ground up to assess LLM proficiency in linguistic-related tasks in pt-PT across eight linguistic dimensions, including Language Variety, Culture-bound Semantics, Discourse Analysis, Word Plays, Syntax, Morphology, Lexicology, and Phonetics and Phonology. ALBA is manually constructed by language experts and paired with an LLM-as-a-judge framework for scalable evaluation of pt-PT generated language. Experiments on a diverse set of models reveal performance variability across linguistic dimensions, highlighting the need for comprehensive, variety-sensitive benchmarks that support further development of tools in pt-PT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_26516 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs Vieira, Inês Calvo, Inês Paulo, Iago Furtado, James Ferreira, Rafael Tavares, Diogo Glória-Silva, Diogo Semedo, David Magalhães, João Computation and Language Artificial Intelligence Machine Learning I.2.7 As Large Language Models (LLMs) expand across multilingual domains, evaluating their performance in under-represented languages becomes increasingly important. European Portuguese (pt-PT) is particularly affected, as existing training data and benchmarks are mainly in Brazilian Portuguese (pt-BR). To address this, we introduce ALBA, a linguistically grounded benchmark designed from the ground up to assess LLM proficiency in linguistic-related tasks in pt-PT across eight linguistic dimensions, including Language Variety, Culture-bound Semantics, Discourse Analysis, Word Plays, Syntax, Morphology, Lexicology, and Phonetics and Phonology. ALBA is manually constructed by language experts and paired with an LLM-as-a-judge framework for scalable evaluation of pt-PT generated language. Experiments on a diverse set of models reveal performance variability across linguistic dimensions, highlighting the need for comprehensive, variety-sensitive benchmarks that support further development of tools in pt-PT. |
| title | ALBA: A European Portuguese Benchmark for Evaluating Language and Linguistic Dimensions in Generative LLMs |
| topic | Computation and Language Artificial Intelligence Machine Learning I.2.7 |
| url | https://arxiv.org/abs/2603.26516 |