Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2510.01474 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917256195211264 |
|---|---|
| author | Marino, Bill Hunter, Rosco Schnabl, Christoph Jamali, Zubair Kalpakos, Marinos Emmanouil Kashyap, Mudra Hinton, Isaiah Hanson, Alexa Nazir, Maahum Steffek, Felix Wen, Hongkai Lane, Nicholas D. |
| author_facet | Marino, Bill Hunter, Rosco Schnabl, Christoph Jamali, Zubair Kalpakos, Marinos Emmanouil Kashyap, Mudra Hinton, Isaiah Hanson, Alexa Nazir, Maahum Steffek, Felix Wen, Hongkai Lane, Nicholas D. |
| contents | As governments move to regulate AI, there is growing interest in using Large Language Models (LLMs) to assess whether or not an AI system complies with a given AI Regulation (AIR). However, there is presently no way to benchmark the performance of LLMs at this task. To fill this void, we introduce AIReg-Bench: the first open benchmark dataset designed to test how well LLMs can assess compliance with the EU AI Act (AIA). We created this dataset through a two-step process: (1) by prompting an LLM with carefully structured instructions, we generated 120 technical documentation excerpts (samples), each depicting a fictional, albeit plausible, AI system -- of the kind an AI provider might produce to demonstrate their compliance with AIR; (2) legal experts then reviewed and annotated each sample to indicate whether, and in what way, the AI system described therein violates specific Articles of the AIA. The resulting dataset, together with our evaluation of whether frontier LLMs can reproduce the experts' compliance labels, provides a starting point to understand the opportunities and limitations of LLM-based AIR compliance assessment tools and establishes a benchmark against which subsequent LLMs can be compared. The dataset and evaluation code are available at https://github.com/camlsys/aireg-bench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_01474 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance Marino, Bill Hunter, Rosco Schnabl, Christoph Jamali, Zubair Kalpakos, Marinos Emmanouil Kashyap, Mudra Hinton, Isaiah Hanson, Alexa Nazir, Maahum Steffek, Felix Wen, Hongkai Lane, Nicholas D. Artificial Intelligence As governments move to regulate AI, there is growing interest in using Large Language Models (LLMs) to assess whether or not an AI system complies with a given AI Regulation (AIR). However, there is presently no way to benchmark the performance of LLMs at this task. To fill this void, we introduce AIReg-Bench: the first open benchmark dataset designed to test how well LLMs can assess compliance with the EU AI Act (AIA). We created this dataset through a two-step process: (1) by prompting an LLM with carefully structured instructions, we generated 120 technical documentation excerpts (samples), each depicting a fictional, albeit plausible, AI system -- of the kind an AI provider might produce to demonstrate their compliance with AIR; (2) legal experts then reviewed and annotated each sample to indicate whether, and in what way, the AI system described therein violates specific Articles of the AIA. The resulting dataset, together with our evaluation of whether frontier LLMs can reproduce the experts' compliance labels, provides a starting point to understand the opportunities and limitations of LLM-based AIR compliance assessment tools and establishes a benchmark against which subsequent LLMs can be compared. The dataset and evaluation code are available at https://github.com/camlsys/aireg-bench. |
| title | AIReg-Bench: Benchmarking Language Models That Assess AI Regulation Compliance |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.01474 |