Cover Image

Challenging the Abilities of Large Language Models in Italian: a Community Initiative

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Nissim, Malvina, Croce, Danilo, Patti, Viviana, Basile, Pierpaolo, Attanasio, Giuseppe, Musacchio, Elio, Rinaldi, Matteo, Borazio, Federico, Francis, Maria, Gili, Jacopo, Scalena, Daniel, Altuna, Begoña, Azurmendi, Ekhi, Basile, Valerio, Bentivogli, Luisa, Bisazza, Arianna, Bolognesi, Marianna, Brunato, Dominique, Caselli, Tommaso, Casola, Silvia, Cassese, Maria, Cettolo, Mauro, Collacciani, Claudia, De Cosmo, Leonardo, Di Buono, Maria Pia, Esuli, Andrea, Etxaniz, Julen, Ferrando, Chiara, Fidelangeli, Alessia, Frenda, Simona, Fusco, Achille, Gaido, Marco, Galassi, Andrea, Galli, Federico, Giordano, Luca, Goffetti, Mattia, Gonzalez-Dios, Itziar, Gregori, Lorenzo, Grundler, Giulia, Iannaccone, Sandro, Jiang, Chunyang, La Quatra, Moreno, Lagioia, Francesca, Lo, Soda Marem, Madeddu, Marco, Magnini, Bernardo, Manna, Raffaele, Mercorio, Fabio, Merlo, Paola, Muti, Arianna, Nastase, Vivi, Negri, Matteo, Onorati, Dario, Palmieri, Elena, Papi, Sara, Passaro, Lucia, Pensa, Giulia, Piergentili, Andrea, Potertì, Daniele, Puccetti, Giovanni, Ranaldi, Federico, Ranaldi, Leonardo, Ravelli, Andrea Amelio, Rosola, Martina, Ruzzetti, Elena Sofia, Samo, Giuseppe, Santilli, Andrea, Santin, Piera, Sarti, Gabriele, Sartor, Giovanni, Savoldi, Beatrice, Serino, Antonio, Seveso, Andrea, Siciliani, Lucia, Torroni, Paolo, Varvara, Rossella, Zaninello, Andrea, Zanollo, Asya, Zanzotto, Fabio Massimo, Zeinalipour, Kamyar, Zugarini, Andrea
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866917124926078976
author	Nissim, Malvina Croce, Danilo Patti, Viviana Basile, Pierpaolo Attanasio, Giuseppe Musacchio, Elio Rinaldi, Matteo Borazio, Federico Francis, Maria Gili, Jacopo Scalena, Daniel Altuna, Begoña Azurmendi, Ekhi Basile, Valerio Bentivogli, Luisa Bisazza, Arianna Bolognesi, Marianna Brunato, Dominique Caselli, Tommaso Casola, Silvia Cassese, Maria Cettolo, Mauro Collacciani, Claudia De Cosmo, Leonardo Di Buono, Maria Pia Esuli, Andrea Etxaniz, Julen Ferrando, Chiara Fidelangeli, Alessia Frenda, Simona Fusco, Achille Gaido, Marco Galassi, Andrea Galli, Federico Giordano, Luca Goffetti, Mattia Gonzalez-Dios, Itziar Gregori, Lorenzo Grundler, Giulia Iannaccone, Sandro Jiang, Chunyang La Quatra, Moreno Lagioia, Francesca Lo, Soda Marem Madeddu, Marco Magnini, Bernardo Manna, Raffaele Mercorio, Fabio Merlo, Paola Muti, Arianna Nastase, Vivi Negri, Matteo Onorati, Dario Palmieri, Elena Papi, Sara Passaro, Lucia Pensa, Giulia Piergentili, Andrea Potertì, Daniele Puccetti, Giovanni Ranaldi, Federico Ranaldi, Leonardo Ravelli, Andrea Amelio Rosola, Martina Ruzzetti, Elena Sofia Samo, Giuseppe Santilli, Andrea Santin, Piera Sarti, Gabriele Sartor, Giovanni Savoldi, Beatrice Serino, Antonio Seveso, Andrea Siciliani, Lucia Torroni, Paolo Varvara, Rossella Zaninello, Andrea Zanollo, Asya Zanzotto, Fabio Massimo Zeinalipour, Kamyar Zugarini, Andrea
author_facet	Nissim, Malvina Croce, Danilo Patti, Viviana Basile, Pierpaolo Attanasio, Giuseppe Musacchio, Elio Rinaldi, Matteo Borazio, Federico Francis, Maria Gili, Jacopo Scalena, Daniel Altuna, Begoña Azurmendi, Ekhi Basile, Valerio Bentivogli, Luisa Bisazza, Arianna Bolognesi, Marianna Brunato, Dominique Caselli, Tommaso Casola, Silvia Cassese, Maria Cettolo, Mauro Collacciani, Claudia De Cosmo, Leonardo Di Buono, Maria Pia Esuli, Andrea Etxaniz, Julen Ferrando, Chiara Fidelangeli, Alessia Frenda, Simona Fusco, Achille Gaido, Marco Galassi, Andrea Galli, Federico Giordano, Luca Goffetti, Mattia Gonzalez-Dios, Itziar Gregori, Lorenzo Grundler, Giulia Iannaccone, Sandro Jiang, Chunyang La Quatra, Moreno Lagioia, Francesca Lo, Soda Marem Madeddu, Marco Magnini, Bernardo Manna, Raffaele Mercorio, Fabio Merlo, Paola Muti, Arianna Nastase, Vivi Negri, Matteo Onorati, Dario Palmieri, Elena Papi, Sara Passaro, Lucia Pensa, Giulia Piergentili, Andrea Potertì, Daniele Puccetti, Giovanni Ranaldi, Federico Ranaldi, Leonardo Ravelli, Andrea Amelio Rosola, Martina Ruzzetti, Elena Sofia Samo, Giuseppe Santilli, Andrea Santin, Piera Sarti, Gabriele Sartor, Giovanni Savoldi, Beatrice Serino, Antonio Seveso, Andrea Siciliani, Lucia Torroni, Paolo Varvara, Rossella Zaninello, Andrea Zanollo, Asya Zanzotto, Fabio Massimo Zeinalipour, Kamyar Zugarini, Andrea
contents	The rapid progress of Large Language Models (LLMs) has transformed natural language processing and broadened its impact across research and society. Yet, systematic evaluation of these models, especially for languages beyond English, remains limited. "Challenging the Abilities of LAnguage Models in ITAlian" (CALAMITA) is a large-scale collaborative benchmarking initiative for Italian, coordinated under the Italian Association for Computational Linguistics. Unlike existing efforts that focus on leaderboards, CALAMITA foregrounds methodology: it federates more than 80 contributors from academia, industry, and the public sector to design, document, and evaluate a diverse collection of tasks, covering linguistic competence, commonsense reasoning, factual consistency, fairness, summarization, translation, and code generation. Through this process, we not only assembled a benchmark of over 20 tasks and almost 100 subtasks, but also established a centralized evaluation pipeline that supports heterogeneous datasets and metrics. We report results for four open-weight LLMs, highlighting systematic strengths and weaknesses across abilities, as well as challenges in task-specific evaluation. Beyond quantitative results, CALAMITA exposes methodological lessons: the necessity of fine-grained, task-representative metrics, the importance of harmonized pipelines, and the benefits and limitations of broad community engagement. CALAMITA is conceived as a rolling benchmark, enabling continuous integration of new tasks and models. This makes it both a resource -- the most comprehensive and diverse benchmark for Italian to date -- and a framework for sustainable, community-driven evaluation. We argue that this combination offers a blueprint for other languages and communities seeking inclusive and rigorous LLM evaluation practices.
format	Preprint
id	arxiv_https___arxiv_org_abs_2512_04759
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Challenging the Abilities of Large Language Models in Italian: a Community Initiative Nissim, Malvina Croce, Danilo Patti, Viviana Basile, Pierpaolo Attanasio, Giuseppe Musacchio, Elio Rinaldi, Matteo Borazio, Federico Francis, Maria Gili, Jacopo Scalena, Daniel Altuna, Begoña Azurmendi, Ekhi Basile, Valerio Bentivogli, Luisa Bisazza, Arianna Bolognesi, Marianna Brunato, Dominique Caselli, Tommaso Casola, Silvia Cassese, Maria Cettolo, Mauro Collacciani, Claudia De Cosmo, Leonardo Di Buono, Maria Pia Esuli, Andrea Etxaniz, Julen Ferrando, Chiara Fidelangeli, Alessia Frenda, Simona Fusco, Achille Gaido, Marco Galassi, Andrea Galli, Federico Giordano, Luca Goffetti, Mattia Gonzalez-Dios, Itziar Gregori, Lorenzo Grundler, Giulia Iannaccone, Sandro Jiang, Chunyang La Quatra, Moreno Lagioia, Francesca Lo, Soda Marem Madeddu, Marco Magnini, Bernardo Manna, Raffaele Mercorio, Fabio Merlo, Paola Muti, Arianna Nastase, Vivi Negri, Matteo Onorati, Dario Palmieri, Elena Papi, Sara Passaro, Lucia Pensa, Giulia Piergentili, Andrea Potertì, Daniele Puccetti, Giovanni Ranaldi, Federico Ranaldi, Leonardo Ravelli, Andrea Amelio Rosola, Martina Ruzzetti, Elena Sofia Samo, Giuseppe Santilli, Andrea Santin, Piera Sarti, Gabriele Sartor, Giovanni Savoldi, Beatrice Serino, Antonio Seveso, Andrea Siciliani, Lucia Torroni, Paolo Varvara, Rossella Zaninello, Andrea Zanollo, Asya Zanzotto, Fabio Massimo Zeinalipour, Kamyar Zugarini, Andrea Computation and Language The rapid progress of Large Language Models (LLMs) has transformed natural language processing and broadened its impact across research and society. Yet, systematic evaluation of these models, especially for languages beyond English, remains limited. "Challenging the Abilities of LAnguage Models in ITAlian" (CALAMITA) is a large-scale collaborative benchmarking initiative for Italian, coordinated under the Italian Association for Computational Linguistics. Unlike existing efforts that focus on leaderboards, CALAMITA foregrounds methodology: it federates more than 80 contributors from academia, industry, and the public sector to design, document, and evaluate a diverse collection of tasks, covering linguistic competence, commonsense reasoning, factual consistency, fairness, summarization, translation, and code generation. Through this process, we not only assembled a benchmark of over 20 tasks and almost 100 subtasks, but also established a centralized evaluation pipeline that supports heterogeneous datasets and metrics. We report results for four open-weight LLMs, highlighting systematic strengths and weaknesses across abilities, as well as challenges in task-specific evaluation. Beyond quantitative results, CALAMITA exposes methodological lessons: the necessity of fine-grained, task-representative metrics, the importance of harmonized pipelines, and the benefits and limitations of broad community engagement. CALAMITA is conceived as a rolling benchmark, enabling continuous integration of new tasks and models. This makes it both a resource -- the most comprehensive and diverse benchmark for Italian to date -- and a framework for sustainable, community-driven evaluation. We argue that this combination offers a blueprint for other languages and communities seeking inclusive and rigorous LLM evaluation practices.
title	Challenging the Abilities of Large Language Models in Italian: a Community Initiative
topic	Computation and Language
url	https://arxiv.org/abs/2512.04759