Relying on the Metrics of Evaluated Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909367209558016 |
|---|---|
| author | Wang, Serena Jordan, Michael I. Ligett, Katrina McAfee, R. Preston |
| author_facet | Wang, Serena Jordan, Michael I. Ligett, Katrina McAfee, R. Preston |
| contents | Online platforms and regulators face a continuing problem of designing effective evaluation metrics. While tools for collecting and processing data continue to progress, this has not addressed the problem of "unknown unknowns", or fundamental informational limitations on part of the evaluator. To guide the choice of metrics in the face of this informational problem, we turn to the evaluated agents themselves, who may have more information about how to measure their own outcomes. We model this interaction as an agency game, where we ask: "When does an agent have an incentive to reveal the observability of a metric to their evaluator?" We show that an agent will prefer to reveal metrics that differentiate the most difficult tasks from the rest, and conceal metrics that differentiate the easiest. We further show that the agent can prefer to reveal a metric "garbled" with noise over both fully concealing and fully revealing. This indicates an economic value to privacy that yields Pareto improvement for both the agent and evaluator. We demonstrate these findings on data from online rideshare platforms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_14005 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Relying on the Metrics of Evaluated Agents Wang, Serena Jordan, Michael I. Ligett, Katrina McAfee, R. Preston Computer Science and Game Theory Theoretical Economics Online platforms and regulators face a continuing problem of designing effective evaluation metrics. While tools for collecting and processing data continue to progress, this has not addressed the problem of "unknown unknowns", or fundamental informational limitations on part of the evaluator. To guide the choice of metrics in the face of this informational problem, we turn to the evaluated agents themselves, who may have more information about how to measure their own outcomes. We model this interaction as an agency game, where we ask: "When does an agent have an incentive to reveal the observability of a metric to their evaluator?" We show that an agent will prefer to reveal metrics that differentiate the most difficult tasks from the rest, and conceal metrics that differentiate the easiest. We further show that the agent can prefer to reveal a metric "garbled" with noise over both fully concealing and fully revealing. This indicates an economic value to privacy that yields Pareto improvement for both the agent and evaluator. We demonstrate these findings on data from online rideshare platforms. |
| title | Relying on the Metrics of Evaluated Agents |
| topic | Computer Science and Game Theory Theoretical Economics |
| url | https://arxiv.org/abs/2402.14005 |