"Trust me on this" Explaining Agent Behavior to a Human Terminator
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915272227553280 |
|---|---|
| author | Menkes, Uri Hallak, Assaf Amir, Ofra |
| author_facet | Menkes, Uri Hallak, Assaf Amir, Ofra |
| contents | Consider a setting where a pre-trained agent is operating in an environment and a human operator can decide to temporarily terminate its operation and take-over for some duration of time. These kind of scenarios are common in human-machine interactions, for example in autonomous driving, factory automation and healthcare. In these settings, we typically observe a trade-off between two extreme cases -- if no take-overs are allowed, then the agent might employ a sub-optimal, possibly dangerous policy. Alternatively, if there are too many take-overs, then the human has no confidence in the agent, greatly limiting its usefulness. In this paper, we formalize this setup and propose an explainability scheme to help optimize the number of human interventions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_04592 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | "Trust me on this" Explaining Agent Behavior to a Human Terminator Menkes, Uri Hallak, Assaf Amir, Ofra Human-Computer Interaction Artificial Intelligence Consider a setting where a pre-trained agent is operating in an environment and a human operator can decide to temporarily terminate its operation and take-over for some duration of time. These kind of scenarios are common in human-machine interactions, for example in autonomous driving, factory automation and healthcare. In these settings, we typically observe a trade-off between two extreme cases -- if no take-overs are allowed, then the agent might employ a sub-optimal, possibly dangerous policy. Alternatively, if there are too many take-overs, then the human has no confidence in the agent, greatly limiting its usefulness. In this paper, we formalize this setup and propose an explainability scheme to help optimize the number of human interventions. |
| title | "Trust me on this" Explaining Agent Behavior to a Human Terminator |
| topic | Human-Computer Interaction Artificial Intelligence |
| url | https://arxiv.org/abs/2504.04592 |