Secure human oversight of AI: Threat modeling in a socio-technical context
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917315570827264 |
|---|---|
| author | Ditz, Jonas C. Lazar, Veronika Lichtmeß, Elmar Plesch, Carola Heck, Matthias Baum, Kevin Langer, Markus |
| author_facet | Ditz, Jonas C. Lazar, Veronika Lichtmeß, Elmar Plesch, Carola Heck, Matthias Baum, Kevin Langer, Markus |
| contents | Human oversight of AI is promoted as a safeguard against risks such as inaccurate outputs, system malfunctions, or violations of fundamental rights, and is mandated in regulation like the European AI Act. Yet debates on human oversight have largely focused on its effectiveness, while overlooking a critical dimension: the security of human oversight. We argue that human oversight creates a new attack surface within the safety, security, and accountability architecture of AI operations. Drawing on cybersecurity perspectives, we model human oversight as an IT application for the purpose of systematic threat modeling of the human oversight process. Threat modeling allows us to identify security risks within human oversight and points towards possible mitigation strategies. Our contributions are: (1) introducing a security perspective on human oversight, (2) offering researchers and practitioners guidance on how to approach their human oversight applications from a security point of view, and (3) providing a systematic overview of attack vectors and hardening strategies to enable secure human oversight of AI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12290 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Secure human oversight of AI: Threat modeling in a socio-technical context Ditz, Jonas C. Lazar, Veronika Lichtmeß, Elmar Plesch, Carola Heck, Matthias Baum, Kevin Langer, Markus Cryptography and Security Computers and Society Human-Computer Interaction Human oversight of AI is promoted as a safeguard against risks such as inaccurate outputs, system malfunctions, or violations of fundamental rights, and is mandated in regulation like the European AI Act. Yet debates on human oversight have largely focused on its effectiveness, while overlooking a critical dimension: the security of human oversight. We argue that human oversight creates a new attack surface within the safety, security, and accountability architecture of AI operations. Drawing on cybersecurity perspectives, we model human oversight as an IT application for the purpose of systematic threat modeling of the human oversight process. Threat modeling allows us to identify security risks within human oversight and points towards possible mitigation strategies. Our contributions are: (1) introducing a security perspective on human oversight, (2) offering researchers and practitioners guidance on how to approach their human oversight applications from a security point of view, and (3) providing a systematic overview of attack vectors and hardening strategies to enable secure human oversight of AI. |
| title | Secure human oversight of AI: Threat modeling in a socio-technical context |
| topic | Cryptography and Security Computers and Society Human-Computer Interaction |
| url | https://arxiv.org/abs/2509.12290 |