Toward an Evaluation Science for Generative AI Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910872394268672 |
|---|---|
| author | Weidinger, Laura Raji, Inioluwa Deborah Wallach, Hanna Mitchell, Margaret Wang, Angelina Salaudeen, Olawale Bommasani, Rishi Ganguli, Deep Koyejo, Sanmi Isaac, William |
| author_facet | Weidinger, Laura Raji, Inioluwa Deborah Wallach, Hanna Mitchell, Margaret Wang, Angelina Salaudeen, Olawale Bommasani, Rishi Ganguli, Deep Koyejo, Sanmi Isaac, William |
| contents | There is an increasing imperative to anticipate and understand the performance and safety of generative AI systems in real-world deployment contexts. However, the current evaluation ecosystem is insufficient: Commonly used static benchmarks face validity challenges, and ad hoc case-by-case audits rarely scale. In this piece, we advocate for maturing an evaluation science for generative AI systems. While generative AI creates unique challenges for system safety engineering and measurement science, the field can draw valuable insights from the development of safety evaluation practices in other fields, including transportation, aerospace, and pharmaceutical engineering. In particular, we present three key lessons: Evaluation metrics must be applicable to real-world performance, metrics must be iteratively refined, and evaluation institutions and norms must be established. Applying these insights, we outline a concrete path toward a more rigorous approach for evaluating generative AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_05336 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Toward an Evaluation Science for Generative AI Systems Weidinger, Laura Raji, Inioluwa Deborah Wallach, Hanna Mitchell, Margaret Wang, Angelina Salaudeen, Olawale Bommasani, Rishi Ganguli, Deep Koyejo, Sanmi Isaac, William Artificial Intelligence Machine Learning There is an increasing imperative to anticipate and understand the performance and safety of generative AI systems in real-world deployment contexts. However, the current evaluation ecosystem is insufficient: Commonly used static benchmarks face validity challenges, and ad hoc case-by-case audits rarely scale. In this piece, we advocate for maturing an evaluation science for generative AI systems. While generative AI creates unique challenges for system safety engineering and measurement science, the field can draw valuable insights from the development of safety evaluation practices in other fields, including transportation, aerospace, and pharmaceutical engineering. In particular, we present three key lessons: Evaluation metrics must be applicable to real-world performance, metrics must be iteratively refined, and evaluation institutions and norms must be established. Applying these insights, we outline a concrete path toward a more rigorous approach for evaluating generative AI systems. |
| title | Toward an Evaluation Science for Generative AI Systems |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2503.05336 |