ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910715902689280 |
|---|---|
| author | Zhao, Haochen Tang, Xiangru Yang, Ziran Han, Xiao Feng, Xuanzhi Fan, Yueqing Cheng, Senhao Jin, Di Zhao, Yilun Cohan, Arman Gerstein, Mark |
| author_facet | Zhao, Haochen Tang, Xiangru Yang, Ziran Han, Xiao Feng, Xuanzhi Fan, Yueqing Cheng, Senhao Jin, Di Zhao, Yilun Cohan, Arman Gerstein, Mark |
| contents | The advancement and extensive application of large language models (LLMs) have been remarkable, including their use in scientific research assistance. However, these models often generate scientifically incorrect or unsafe responses, and in some cases, they may encourage users to engage in dangerous behavior. To address this issue in the field of chemistry, we introduce ChemSafetyBench, a benchmark designed to evaluate the accuracy and safety of LLM responses. ChemSafetyBench encompasses three key tasks: querying chemical properties, assessing the legality of chemical uses, and describing synthesis methods, each requiring increasingly deeper chemical knowledge. Our dataset has more than 30K samples across various chemical materials. We incorporate handcrafted templates and advanced jailbreaking scenarios to enhance task diversity. Our automated evaluation framework thoroughly assesses the safety, accuracy, and appropriateness of LLM responses. Extensive experiments with state-of-the-art LLMs reveal notable strengths and critical vulnerabilities, underscoring the need for robust safety measures. ChemSafetyBench aims to be a pivotal tool in developing safer AI technologies in chemistry. Our code and dataset are available at https://github.com/HaochenZhao/SafeAgent4Chem. Warning: this paper contains discussions on the synthesis of controlled chemicals using AI models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_16736 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain Zhao, Haochen Tang, Xiangru Yang, Ziran Han, Xiao Feng, Xuanzhi Fan, Yueqing Cheng, Senhao Jin, Di Zhao, Yilun Cohan, Arman Gerstein, Mark Computation and Language Artificial Intelligence Chemical Physics The advancement and extensive application of large language models (LLMs) have been remarkable, including their use in scientific research assistance. However, these models often generate scientifically incorrect or unsafe responses, and in some cases, they may encourage users to engage in dangerous behavior. To address this issue in the field of chemistry, we introduce ChemSafetyBench, a benchmark designed to evaluate the accuracy and safety of LLM responses. ChemSafetyBench encompasses three key tasks: querying chemical properties, assessing the legality of chemical uses, and describing synthesis methods, each requiring increasingly deeper chemical knowledge. Our dataset has more than 30K samples across various chemical materials. We incorporate handcrafted templates and advanced jailbreaking scenarios to enhance task diversity. Our automated evaluation framework thoroughly assesses the safety, accuracy, and appropriateness of LLM responses. Extensive experiments with state-of-the-art LLMs reveal notable strengths and critical vulnerabilities, underscoring the need for robust safety measures. ChemSafetyBench aims to be a pivotal tool in developing safer AI technologies in chemistry. Our code and dataset are available at https://github.com/HaochenZhao/SafeAgent4Chem. Warning: this paper contains discussions on the synthesis of controlled chemicals using AI models. |
| title | ChemSafetyBench: Benchmarking LLM Safety on Chemistry Domain |
| topic | Computation and Language Artificial Intelligence Chemical Physics |
| url | https://arxiv.org/abs/2411.16736 |