Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimators
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917750398517248 |
|---|---|
| author | Towers, Mark Du, Yali Freeman, Christopher Norman, Timothy J. |
| author_facet | Towers, Mark Du, Yali Freeman, Christopher Norman, Timothy J. |
| contents | Future reward estimation is a core component of reinforcement learning agents; i.e., Q-value and state-value functions, predicting an agent's sum of future rewards. Their scalar output, however, obfuscates when or what individual future rewards an agent may expect to receive. We address this by modifying an agent's future reward estimator to predict their next N expected rewards, referred to as Temporal Reward Decomposition (TRD). This unlocks novel explanations of agent behaviour. Through TRD we can: estimate when an agent may expect to receive a reward, the value of the reward and the agent's confidence in receiving it; measure an input feature's temporal importance to the agent's action decisions; and predict the influence of different actions on future rewards. Furthermore, we show that DQN agents trained on Atari environments can be efficiently retrained to incorporate TRD with minimal impact on performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_08230 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimators Towers, Mark Du, Yali Freeman, Christopher Norman, Timothy J. Artificial Intelligence Machine Learning Future reward estimation is a core component of reinforcement learning agents; i.e., Q-value and state-value functions, predicting an agent's sum of future rewards. Their scalar output, however, obfuscates when or what individual future rewards an agent may expect to receive. We address this by modifying an agent's future reward estimator to predict their next N expected rewards, referred to as Temporal Reward Decomposition (TRD). This unlocks novel explanations of agent behaviour. Through TRD we can: estimate when an agent may expect to receive a reward, the value of the reward and the agent's confidence in receiving it; measure an input feature's temporal importance to the agent's action decisions; and predict the influence of different actions on future rewards. Furthermore, we show that DQN agents trained on Atari environments can be efficiently retrained to incorporate TRD with minimal impact on performance. |
| title | Explaining an Agent's Future Beliefs through Temporally Decomposing Future Reward Estimators |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2408.08230 |