Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910118475464704 |
|---|---|
| author | Mittal, Avni Kumar, Shanu Dandapat, Sandipan Choudhury, Monojit |
| author_facet | Mittal, Avni Kumar, Shanu Dandapat, Sandipan Choudhury, Monojit |
| contents | We study predictive multilingual evaluation: estimating how well a model will perform on a task in a target language when direct benchmark results are missing. This problem is common in multilingual deployment, where evaluation coverage is sparse and published evidence is uneven across languages, tasks, and model families. We introduce a controlled benchmark of 1,500 questions spanning six tasks and five evidence scenarios. The benchmark separates accessible evidence from ground truth, enabling evaluation of systems that must infer missing results from incomplete literature evidence. We also present Litmus (Re)Agent, a DAG-orchestrated agentic system that decomposes queries into hypotheses, retrieves evidence, and synthesises predictions through feature-aware aggregation. Across six systems, Litmus (Re)Agent achieves the best overall performance, with the largest gains in transfer-heavy scenarios where direct evidence is weak or absent. These results show that structured agentic reasoning is a promising approach to multilingual performance estimation under incomplete evidence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08970 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models Mittal, Avni Kumar, Shanu Dandapat, Sandipan Choudhury, Monojit Computation and Language Artificial Intelligence Human-Computer Interaction Multiagent Systems We study predictive multilingual evaluation: estimating how well a model will perform on a task in a target language when direct benchmark results are missing. This problem is common in multilingual deployment, where evaluation coverage is sparse and published evidence is uneven across languages, tasks, and model families. We introduce a controlled benchmark of 1,500 questions spanning six tasks and five evidence scenarios. The benchmark separates accessible evidence from ground truth, enabling evaluation of systems that must infer missing results from incomplete literature evidence. We also present Litmus (Re)Agent, a DAG-orchestrated agentic system that decomposes queries into hypotheses, retrieves evidence, and synthesises predictions through feature-aware aggregation. Across six systems, Litmus (Re)Agent achieves the best overall performance, with the largest gains in transfer-heavy scenarios where direct evidence is weak or absent. These results show that structured agentic reasoning is a promising approach to multilingual performance estimation under incomplete evidence. |
| title | Litmus (Re)Agent: A Benchmark and Agentic System for Predictive Evaluation of Multilingual Models |
| topic | Computation and Language Artificial Intelligence Human-Computer Interaction Multiagent Systems |
| url | https://arxiv.org/abs/2604.08970 |