TRIAGE: Ethical Benchmarking of AI Models Through Mass Casualty Simulations
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910683290927104 |
|---|---|
| author | Kirch, Nathalie Maria Hebenstreit, Konstantin Samwald, Matthias |
| author_facet | Kirch, Nathalie Maria Hebenstreit, Konstantin Samwald, Matthias |
| contents | We present the TRIAGE Benchmark, a novel machine ethics (ME) benchmark that tests LLMs' ability to make ethical decisions during mass casualty incidents. It uses real-world ethical dilemmas with clear solutions designed by medical professionals, offering a more realistic alternative to annotation-based benchmarks. TRIAGE incorporates various prompting styles to evaluate model performance across different contexts. Most models consistently outperformed random guessing, suggesting LLMs may support decision-making in triage scenarios. Neutral or factual scenario formulations led to the best performance, unlike other ME benchmarks where ethical reminders improved outcomes. Adversarial prompts reduced performance but not to random guessing levels. Open-source models made more morally serious errors, and general capability overall predicted better performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_18991 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TRIAGE: Ethical Benchmarking of AI Models Through Mass Casualty Simulations Kirch, Nathalie Maria Hebenstreit, Konstantin Samwald, Matthias Computers and Society Artificial Intelligence We present the TRIAGE Benchmark, a novel machine ethics (ME) benchmark that tests LLMs' ability to make ethical decisions during mass casualty incidents. It uses real-world ethical dilemmas with clear solutions designed by medical professionals, offering a more realistic alternative to annotation-based benchmarks. TRIAGE incorporates various prompting styles to evaluate model performance across different contexts. Most models consistently outperformed random guessing, suggesting LLMs may support decision-making in triage scenarios. Neutral or factual scenario formulations led to the best performance, unlike other ME benchmarks where ethical reminders improved outcomes. Adversarial prompts reduced performance but not to random guessing levels. Open-source models made more morally serious errors, and general capability overall predicted better performance. |
| title | TRIAGE: Ethical Benchmarking of AI Models Through Mass Casualty Simulations |
| topic | Computers and Society Artificial Intelligence |
| url | https://arxiv.org/abs/2410.18991 |