Evalita-LLM: Benchmarking Large Language Models on Italian
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913677990428672 |
|---|---|
| author | Magnini, Bernardo Zanoli, Roberto Resta, Michele Cimmino, Martin Albano, Paolo Madeddu, Marco Patti, Viviana |
| author_facet | Magnini, Bernardo Zanoli, Roberto Resta, Michele Cimmino, Martin Albano, Paolo Madeddu, Marco Patti, Viviana |
| contents | We describe Evalita-LLM, a new benchmark designed to evaluate Large Language Models (LLMs) on Italian tasks. The distinguishing and innovative features of Evalita-LLM are the following: (i) all tasks are native Italian, avoiding issues of translating from Italian and potential cultural biases; (ii) in addition to well established multiple-choice tasks, the benchmark includes generative tasks, enabling more natural interaction with LLMs; (iii) all tasks are evaluated against multiple prompts, this way mitigating the model sensitivity to specific prompts and allowing a fairer and objective evaluation. We propose an iterative methodology, where candidate tasks and candidate prompts are validated against a set of LLMs used for development. We report experimental results from the benchmark's development phase, and provide performance statistics for several state-of-the-art LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02289 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evalita-LLM: Benchmarking Large Language Models on Italian Magnini, Bernardo Zanoli, Roberto Resta, Michele Cimmino, Martin Albano, Paolo Madeddu, Marco Patti, Viviana Computation and Language We describe Evalita-LLM, a new benchmark designed to evaluate Large Language Models (LLMs) on Italian tasks. The distinguishing and innovative features of Evalita-LLM are the following: (i) all tasks are native Italian, avoiding issues of translating from Italian and potential cultural biases; (ii) in addition to well established multiple-choice tasks, the benchmark includes generative tasks, enabling more natural interaction with LLMs; (iii) all tasks are evaluated against multiple prompts, this way mitigating the model sensitivity to specific prompts and allowing a fairer and objective evaluation. We propose an iterative methodology, where candidate tasks and candidate prompts are validated against a set of LLMs used for development. We report experimental results from the benchmark's development phase, and provide performance statistics for several state-of-the-art LLMs. |
| title | Evalita-LLM: Benchmarking Large Language Models on Italian |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.02289 |