WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910570882531328 |
|---|---|
| author | Gupta, Prannaya Yau, Le Qi Low, Hao Han Lee, I-Shiang Lim, Hugo Maximus Teoh, Yu Xin Koh, Jia Hng Liew, Dar Win Bhardwaj, Rishabh Bhardwaj, Rajat Poria, Soujanya |
| author_facet | Gupta, Prannaya Yau, Le Qi Low, Hao Han Lee, I-Shiang Lim, Hugo Maximus Teoh, Yu Xin Koh, Jia Hng Liew, Dar Win Bhardwaj, Rishabh Bhardwaj, Rajat Poria, Soujanya |
| contents | WalledEval is a comprehensive AI safety testing toolkit designed to evaluate large language models (LLMs). It accommodates a diverse range of models, including both open-weight and API-based ones, and features over 35 safety benchmarks covering areas such as multilingual safety, exaggerated safety, and prompt injections. The framework supports both LLM and judge benchmarking and incorporates custom mutators to test safety against various text-style mutations, such as future tense and paraphrasing. Additionally, WalledEval introduces WalledGuard, a new, small, and performant content moderation tool, and two datasets: SGXSTest and HIXSTest, which serve as benchmarks for assessing the exaggerated safety of LLMs and judges in cultural contexts. We make WalledEval publicly available at https://github.com/walledai/walledeval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_03837 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models Gupta, Prannaya Yau, Le Qi Low, Hao Han Lee, I-Shiang Lim, Hugo Maximus Teoh, Yu Xin Koh, Jia Hng Liew, Dar Win Bhardwaj, Rishabh Bhardwaj, Rajat Poria, Soujanya Computation and Language Artificial Intelligence WalledEval is a comprehensive AI safety testing toolkit designed to evaluate large language models (LLMs). It accommodates a diverse range of models, including both open-weight and API-based ones, and features over 35 safety benchmarks covering areas such as multilingual safety, exaggerated safety, and prompt injections. The framework supports both LLM and judge benchmarking and incorporates custom mutators to test safety against various text-style mutations, such as future tense and paraphrasing. Additionally, WalledEval introduces WalledGuard, a new, small, and performant content moderation tool, and two datasets: SGXSTest and HIXSTest, which serve as benchmarks for assessing the exaggerated safety of LLMs and judges in cultural contexts. We make WalledEval publicly available at https://github.com/walledai/walledeval. |
| title | WalledEval: A Comprehensive Safety Evaluation Toolkit for Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2408.03837 |