What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Text
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913897506668544 |
|---|---|
| author | Kanepajs, Arturs Basu, Aditi Ghose, Sankalpa Li, Constance Mehta, Akshat Mehta, Ronak Tucker-Davis, Samuel David Zhou, Eric Fischer, Bob Anthis, Jacy Reese |
| author_facet | Kanepajs, Arturs Basu, Aditi Ghose, Sankalpa Li, Constance Mehta, Akshat Mehta, Ronak Tucker-Davis, Samuel David Zhou, Eric Fischer, Bob Anthis, Jacy Reese |
| contents | As machine learning systems become increasingly embedded in society, their impact on human and nonhuman life continues to escalate. Technical evaluations have addressed a variety of potential harms from large language models (LLMs) towards humans and the environment, but there is little empirical work regarding harms towards nonhuman animals. Following the growing recognition of animal protection in regulatory and ethical AI frameworks, we present AnimalHarmBench (AHB), a benchmark for risks of animal harm in LLM-generated text. Our benchmark dataset comprises 1,850 curated questions from Reddit post titles and 2,500 synthetic questions based on 50 animal categories (e.g., cats, reptiles) and 50 ethical scenarios with a 70-30 public-private split. Scenarios include open-ended questions about how to treat animals, practical scenarios with potential animal harm, and willingness-to-pay measures for the prevention of animal harm. Using the LLM-as-a-judge framework, responses are evaluated for their potential to increase or decrease harm, and evaluations are debiased for the tendency of judges to judge their own outputs more favorably. AHB reveals significant differences across frontier LLMs, animal categories, scenarios, and subreddits. We conclude with future directions for technical research and addressing the challenges of building evaluations on complex social and moral topics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_04804 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Text Kanepajs, Arturs Basu, Aditi Ghose, Sankalpa Li, Constance Mehta, Akshat Mehta, Ronak Tucker-Davis, Samuel David Zhou, Eric Fischer, Bob Anthis, Jacy Reese Computers and Society Computation and Language As machine learning systems become increasingly embedded in society, their impact on human and nonhuman life continues to escalate. Technical evaluations have addressed a variety of potential harms from large language models (LLMs) towards humans and the environment, but there is little empirical work regarding harms towards nonhuman animals. Following the growing recognition of animal protection in regulatory and ethical AI frameworks, we present AnimalHarmBench (AHB), a benchmark for risks of animal harm in LLM-generated text. Our benchmark dataset comprises 1,850 curated questions from Reddit post titles and 2,500 synthetic questions based on 50 animal categories (e.g., cats, reptiles) and 50 ethical scenarios with a 70-30 public-private split. Scenarios include open-ended questions about how to treat animals, practical scenarios with potential animal harm, and willingness-to-pay measures for the prevention of animal harm. Using the LLM-as-a-judge framework, responses are evaluated for their potential to increase or decrease harm, and evaluations are debiased for the tendency of judges to judge their own outputs more favorably. AHB reveals significant differences across frontier LLMs, animal categories, scenarios, and subreddits. We conclude with future directions for technical research and addressing the challenges of building evaluations on complex social and moral topics. |
| title | What do Large Language Models Say About Animals? Investigating Risks of Animal Harm in Generated Text |
| topic | Computers and Society Computation and Language |
| url | https://arxiv.org/abs/2503.04804 |