Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915974427443200 |
|---|---|
| author | Junior, Roseval Malaquias Bonás, Giovana Kerche Almeida, Thales Sales Abonizio, Hugo Laitz, Thiago Pires, Ramon Piau, Marcos Larcher, Celio Nogueira, Rodrigo |
| author_facet | Junior, Roseval Malaquias Bonás, Giovana Kerche Almeida, Thales Sales Abonizio, Hugo Laitz, Thiago Pires, Ramon Piau, Marcos Larcher, Celio Nogueira, Rodrigo |
| contents | Rankings produced by holistic LLM-as-a-judge scoring are sensitive to the bias of the chosen judge model. We show that switching to binary rubric scoring with multi-judge filtering removes this sensitivity: decomposing the judgement matters more than the judge model itself. To support this claim, we introduce Prosa, the first real user multi-turn Brazilian Portuguese chat benchmark: 1,000 WildChat conversations scored by three judges from three model families on 16 models. Under filtered rubric scoring the three judges agree on every one of the 16 ranks, whereas under holistic scoring they agree on only 7 of 16. Additionally, the rubric filtering pipeline increases the average score gap between neighbouring models by 47%, thereby improving Prosa's discriminative power. Evaluating a new model on Prosa costs approximately $2.1 when using Gemini 3 Flash as the judge. We release the benchmark and the filtering code to ensure that future models can be assessed under identical conditions. These artifacts also make our rubric-based scoring method reusable beyond Prosa, supporting other open-ended evaluation settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_01630 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese Junior, Roseval Malaquias Bonás, Giovana Kerche Almeida, Thales Sales Abonizio, Hugo Laitz, Thiago Pires, Ramon Piau, Marcos Larcher, Celio Nogueira, Rodrigo Computation and Language Artificial Intelligence Rankings produced by holistic LLM-as-a-judge scoring are sensitive to the bias of the chosen judge model. We show that switching to binary rubric scoring with multi-judge filtering removes this sensitivity: decomposing the judgement matters more than the judge model itself. To support this claim, we introduce Prosa, the first real user multi-turn Brazilian Portuguese chat benchmark: 1,000 WildChat conversations scored by three judges from three model families on 16 models. Under filtered rubric scoring the three judges agree on every one of the 16 ranks, whereas under holistic scoring they agree on only 7 of 16. Additionally, the rubric filtering pipeline increases the average score gap between neighbouring models by 47%, thereby improving Prosa's discriminative power. Evaluating a new model on Prosa costs approximately $2.1 when using Gemini 3 Flash as the judge. We release the benchmark and the filtering code to ensure that future models can be assessed under identical conditions. These artifacts also make our rubric-based scoring method reusable beyond Prosa, supporting other open-ended evaluation settings. |
| title | Prosa: Rubric-Based Evaluation of LLMs on Real User Chats in Brazilian Portuguese |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2605.01630 |