PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917107304759296 |
|---|---|
| author | Almeida, Thales Sales Pires, Ramon Abonizio, Hugo Nogueira, Rodrigo Pedrini, Hélio |
| author_facet | Almeida, Thales Sales Pires, Ramon Abonizio, Hugo Nogueira, Rodrigo Pedrini, Hélio |
| contents | Large Language Models (LLMs) exhibit significant variations in performance across linguistic and cultural contexts, underscoring the need for systematic evaluation in diverse languages. In this work, we present the most extensive evaluation of LLMs for the Portuguese language to date. Leveraging our newly introduced PoETa v2 benchmark -- a comprehensive suite of over 40 tasks in Portuguese -- we assess more than 20 models covering a broad spectrum of training scales and computational resources. Our study reveals how computational investment and language-specific adaptation impact performance in Portuguese, while also analyzing performance gaps in comparison to equivalent tasks in English. Through this benchmark and analysis, PoETa v2 lays the groundwork for future research on Portuguese language modeling and evaluation. The benchmark is available at https://github.com/PoETaV2/PoETaV2. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_17808 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese Almeida, Thales Sales Pires, Ramon Abonizio, Hugo Nogueira, Rodrigo Pedrini, Hélio Computation and Language Large Language Models (LLMs) exhibit significant variations in performance across linguistic and cultural contexts, underscoring the need for systematic evaluation in diverse languages. In this work, we present the most extensive evaluation of LLMs for the Portuguese language to date. Leveraging our newly introduced PoETa v2 benchmark -- a comprehensive suite of over 40 tasks in Portuguese -- we assess more than 20 models covering a broad spectrum of training scales and computational resources. Our study reveals how computational investment and language-specific adaptation impact performance in Portuguese, while also analyzing performance gaps in comparison to equivalent tasks in English. Through this benchmark and analysis, PoETa v2 lays the groundwork for future research on Portuguese language modeling and evaluation. The benchmark is available at https://github.com/PoETaV2/PoETaV2. |
| title | PoETa v2: Toward More Robust Evaluation of Large Language Models in Portuguese |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.17808 |