Attention Please: What Transformer Models Really Learn for Process Prediction
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911987322060800 |
|---|---|
| author | Käppel, Martin Ackermann, Lars Jablonski, Stefan Härtl, Simon |
| author_facet | Käppel, Martin Ackermann, Lars Jablonski, Stefan Härtl, Simon |
| contents | Predictive process monitoring aims to support the execution of a process during runtime with various predictions about the further evolution of a process instance. In the last years a plethora of deep learning architectures have been established as state-of-the-art for different prediction targets, among others the transformer architecture. The transformer architecture is equipped with a powerful attention mechanism, assigning attention scores to each input part that allows to prioritize most relevant information leading to more accurate and contextual output. However, deep learning models largely represent a black box, i.e., their reasoning or decision-making process cannot be understood in detail. This paper examines whether the attention scores of a transformer based next-activity prediction model can serve as an explanation for its decision-making. We find that attention scores in next-activity prediction models can serve as explainers and exploit this fact in two proposed graph-based explanation approaches. The gained insights could inspire future work on the improvement of predictive business process models as well as enabling a neural network based mining of process models from event logs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_07097 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Attention Please: What Transformer Models Really Learn for Process Prediction Käppel, Martin Ackermann, Lars Jablonski, Stefan Härtl, Simon Machine Learning Artificial Intelligence 68T07, 68T01, 68U35 H.4.2; I.2.1; I.2.6 Predictive process monitoring aims to support the execution of a process during runtime with various predictions about the further evolution of a process instance. In the last years a plethora of deep learning architectures have been established as state-of-the-art for different prediction targets, among others the transformer architecture. The transformer architecture is equipped with a powerful attention mechanism, assigning attention scores to each input part that allows to prioritize most relevant information leading to more accurate and contextual output. However, deep learning models largely represent a black box, i.e., their reasoning or decision-making process cannot be understood in detail. This paper examines whether the attention scores of a transformer based next-activity prediction model can serve as an explanation for its decision-making. We find that attention scores in next-activity prediction models can serve as explainers and exploit this fact in two proposed graph-based explanation approaches. The gained insights could inspire future work on the improvement of predictive business process models as well as enabling a neural network based mining of process models from event logs. |
| title | Attention Please: What Transformer Models Really Learn for Process Prediction |
| topic | Machine Learning Artificial Intelligence 68T07, 68T01, 68U35 H.4.2; I.2.1; I.2.6 |
| url | https://arxiv.org/abs/2408.07097 |