Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929646545666048 |
|---|---|
| author | Li, Haonan Han, Xudong Zhai, Zenan Mu, Honglin Wang, Hao Zhang, Zhenxuan Geng, Yilin Lin, Shom Wang, Renxi Shelmanov, Artem Qi, Xiangyu Wang, Yuxia Hong, Donghai Yuan, Youliang Chen, Meng Tu, Haoqin Koto, Fajri Kuribayashi, Tatsuki Zeng, Cong Bhardwaj, Rishabh Zhao, Bingchen Duan, Yawen Liu, Yi Alghamdi, Emad A. Yang, Yaodong Dong, Yinpeng Poria, Soujanya Liu, Pengfei Liu, Zhengzhong Ren, Xuguang Hovy, Eduard Gurevych, Iryna Nakov, Preslav Choudhury, Monojit Baldwin, Timothy |
| author_facet | Li, Haonan Han, Xudong Zhai, Zenan Mu, Honglin Wang, Hao Zhang, Zhenxuan Geng, Yilin Lin, Shom Wang, Renxi Shelmanov, Artem Qi, Xiangyu Wang, Yuxia Hong, Donghai Yuan, Youliang Chen, Meng Tu, Haoqin Koto, Fajri Kuribayashi, Tatsuki Zeng, Cong Bhardwaj, Rishabh Zhao, Bingchen Duan, Yawen Liu, Yi Alghamdi, Emad A. Yang, Yaodong Dong, Yinpeng Poria, Soujanya Liu, Pengfei Liu, Zhengzhong Ren, Xuguang Hovy, Eduard Gurevych, Iryna Nakov, Preslav Choudhury, Monojit Baldwin, Timothy |
| contents | To address this gap, we introduce Libra-Leaderboard, a comprehensive framework designed to rank LLMs through a balanced evaluation of performance and safety. Combining a dynamic leaderboard with an interactive LLM arena, Libra-Leaderboard encourages the joint optimization of capability and safety. Unlike traditional approaches that average performance and safety metrics, Libra-Leaderboard uses a distance-to-optimal-score method to calculate the overall rankings. This approach incentivizes models to achieve a balance rather than excelling in one dimension at the expense of some other ones. In the first release, Libra-Leaderboard evaluates 26 mainstream LLMs from 14 leading organizations, identifying critical safety challenges even in state-of-the-art models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_18551 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability Li, Haonan Han, Xudong Zhai, Zenan Mu, Honglin Wang, Hao Zhang, Zhenxuan Geng, Yilin Lin, Shom Wang, Renxi Shelmanov, Artem Qi, Xiangyu Wang, Yuxia Hong, Donghai Yuan, Youliang Chen, Meng Tu, Haoqin Koto, Fajri Kuribayashi, Tatsuki Zeng, Cong Bhardwaj, Rishabh Zhao, Bingchen Duan, Yawen Liu, Yi Alghamdi, Emad A. Yang, Yaodong Dong, Yinpeng Poria, Soujanya Liu, Pengfei Liu, Zhengzhong Ren, Xuguang Hovy, Eduard Gurevych, Iryna Nakov, Preslav Choudhury, Monojit Baldwin, Timothy Computation and Language To address this gap, we introduce Libra-Leaderboard, a comprehensive framework designed to rank LLMs through a balanced evaluation of performance and safety. Combining a dynamic leaderboard with an interactive LLM arena, Libra-Leaderboard encourages the joint optimization of capability and safety. Unlike traditional approaches that average performance and safety metrics, Libra-Leaderboard uses a distance-to-optimal-score method to calculate the overall rankings. This approach incentivizes models to achieve a balance rather than excelling in one dimension at the expense of some other ones. In the first release, Libra-Leaderboard evaluates 26 mainstream LLMs from 14 leading organizations, identifying critical safety challenges even in state-of-the-art models. |
| title | Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capability |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.18551 |