CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908908668321792 |
|---|---|
| author | Bonás, Giovana Kerche Junior, Roseval Malaquias Piau, Marcos Laitz, Thiago Almeida, Thales Sales Abonizio, Hugo Larcher, Celio Pires, Ramon Nogueira, Rodrigo |
| author_facet | Bonás, Giovana Kerche Junior, Roseval Malaquias Piau, Marcos Laitz, Thiago Almeida, Thales Sales Abonizio, Hugo Larcher, Celio Pires, Ramon Nogueira, Rodrigo |
| contents | We introduce CAPITU, a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese. Unlike existing benchmarks that focus on English or use generic prompts, CAPITU contextualizes all tasks within eight canonical works of Brazilian literature, combining verifiable instruction constraints with culturally-grounded content. The benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable without requiring LLM judges or human evaluation. Instruction types include Portuguese-specific linguistic constraints (word termination patterns like -ando/-endo/-indo, -inho/-inha, -mente) and structural requirements. We evaluate 18 state-of-the-art models across single-turn and multi-turn settings. Our results show that frontier reasoning models achieve strong performance (GPT-5.2 with reasoning: 98.5% strict accuracy), while Portuguese-specialized models offer competitive cost-efficiency (Sabiazinho-4: 87.0% at \$0.13 vs Claude-Haiku-4.5: 73.5% at \$1.12). Multi-turn evaluation reveals significant variation in constraint persistence, with conversation-level accuracy ranging from 60% to 96% across models. We identify specific challenges in morphological constraints, exact counting, and constraint persistence degradation across turns. We release the complete benchmark, evaluation code, and baseline results to facilitate research on instruction-following in Portuguese. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_22576 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context Bonás, Giovana Kerche Junior, Roseval Malaquias Piau, Marcos Laitz, Thiago Almeida, Thales Sales Abonizio, Hugo Larcher, Celio Pires, Ramon Nogueira, Rodrigo Computation and Language We introduce CAPITU, a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese. Unlike existing benchmarks that focus on English or use generic prompts, CAPITU contextualizes all tasks within eight canonical works of Brazilian literature, combining verifiable instruction constraints with culturally-grounded content. The benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable without requiring LLM judges or human evaluation. Instruction types include Portuguese-specific linguistic constraints (word termination patterns like -ando/-endo/-indo, -inho/-inha, -mente) and structural requirements. We evaluate 18 state-of-the-art models across single-turn and multi-turn settings. Our results show that frontier reasoning models achieve strong performance (GPT-5.2 with reasoning: 98.5% strict accuracy), while Portuguese-specialized models offer competitive cost-efficiency (Sabiazinho-4: 87.0% at \$0.13 vs Claude-Haiku-4.5: 73.5% at \$1.12). Multi-turn evaluation reveals significant variation in constraint persistence, with conversation-level accuracy ranging from 60% to 96% across models. We identify specific challenges in morphological constraints, exact counting, and constraint persistence degradation across turns. We release the complete benchmark, evaluation code, and baseline results to facilitate research on instruction-following in Portuguese. |
| title | CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.22576 |