MARCA: A Checklist-Based Benchmark for Multilingual Web Search
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915939504619520 |
|---|---|
| author | Almeida, Thales Sales Bonás, Giovana Kerche Pires, Ramon Larcher, Celio Abonizio, Hugo Piau, Marcos Junior, Roseval Malaquias Nogueira, Rodrigo Laitz, Thiago |
| author_facet | Almeida, Thales Sales Bonás, Giovana Kerche Pires, Ramon Larcher, Celio Abonizio, Hugo Piau, Marcos Junior, Roseval Malaquias Nogueira, Rodrigo Laitz, Thiago |
| contents | Large language models (LLMs) are increasingly used as sources of information, yet their reliability depends on the ability to search the web, select relevant evidence, and synthesize complete answers. While recent benchmarks evaluate web-browsing and agentic tool use, multilingual settings, and Portuguese in particular, remain underexplored. We present \textsc{MARCA}, a bilingual (English and Portuguese) benchmark for evaluating LLMs on web-based information seeking. \textsc{MARCA} consists of 52 manually authored multi-entity questions, paired with manually validated checklist-style rubrics that explicitly measure answer completeness and correctness. We evaluate 14 models under two interaction settings: a Basic framework with direct web search and scraping, and an Orchestrator framework that enables task decomposition via delegated subagents. To capture stochasticity, each question is executed multiple times and performance is reported with run-level uncertainty. Across models, we observe large performance differences, find that orchestration often improves coverage, and identify substantial variability in how models transfer from English to Portuguese. The benchmark is available at https://github.com/maritaca-ai/MARCA |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_14448 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MARCA: A Checklist-Based Benchmark for Multilingual Web Search Almeida, Thales Sales Bonás, Giovana Kerche Pires, Ramon Larcher, Celio Abonizio, Hugo Piau, Marcos Junior, Roseval Malaquias Nogueira, Rodrigo Laitz, Thiago Computation and Language Large language models (LLMs) are increasingly used as sources of information, yet their reliability depends on the ability to search the web, select relevant evidence, and synthesize complete answers. While recent benchmarks evaluate web-browsing and agentic tool use, multilingual settings, and Portuguese in particular, remain underexplored. We present \textsc{MARCA}, a bilingual (English and Portuguese) benchmark for evaluating LLMs on web-based information seeking. \textsc{MARCA} consists of 52 manually authored multi-entity questions, paired with manually validated checklist-style rubrics that explicitly measure answer completeness and correctness. We evaluate 14 models under two interaction settings: a Basic framework with direct web search and scraping, and an Orchestrator framework that enables task decomposition via delegated subagents. To capture stochasticity, each question is executed multiple times and performance is reported with run-level uncertainty. Across models, we observe large performance differences, find that orchestration often improves coverage, and identify substantial variability in how models transfer from English to Portuguese. The benchmark is available at https://github.com/maritaca-ai/MARCA |
| title | MARCA: A Checklist-Based Benchmark for Multilingual Web Search |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.14448 |