Unsupervised Log Anomaly Detection with Few Unique Tokens
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916967775993856 |
|---|---|
| author | Sulc, Antonin Eichler, Annika Wilksen, Tim |
| author_facet | Sulc, Antonin Eichler, Annika Wilksen, Tim |
| contents | This article introduces a novel method for detecting anomalies within log data from control system nodes at the European XFEL accelerator. Effective anomaly detection is crucial for providing operators with a clear understanding of each node's availability, status, and potential problems, thereby ensuring smooth accelerator operation. Traditional and learning-based anomaly detection methods face significant limitations due to the sequential nature of these logs and the lack of a rich, node-specific text corpus. To address this, we propose an approach utilizing word embeddings to represent log entries and a Hidden Markov Model (HMM) to model the typical sequential patterns of these embeddings for individual nodes. Anomalies are identified by scoring individual log entries based on a probability ratio: this ratio compares the likelihood of the log sequence including the new entry against its likelihood without it, effectively measuring how well the new entry fits the established pattern. High scores indicate potential anomalies that deviate from the node's routine behavior. This method functions as a warning system, alerting operators to irregular log events that may signify underlying issues, thereby facilitating proactive intervention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_08951 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Unsupervised Log Anomaly Detection with Few Unique Tokens Sulc, Antonin Eichler, Annika Wilksen, Tim Cryptography and Security This article introduces a novel method for detecting anomalies within log data from control system nodes at the European XFEL accelerator. Effective anomaly detection is crucial for providing operators with a clear understanding of each node's availability, status, and potential problems, thereby ensuring smooth accelerator operation. Traditional and learning-based anomaly detection methods face significant limitations due to the sequential nature of these logs and the lack of a rich, node-specific text corpus. To address this, we propose an approach utilizing word embeddings to represent log entries and a Hidden Markov Model (HMM) to model the typical sequential patterns of these embeddings for individual nodes. Anomalies are identified by scoring individual log entries based on a probability ratio: this ratio compares the likelihood of the log sequence including the new entry against its likelihood without it, effectively measuring how well the new entry fits the established pattern. High scores indicate potential anomalies that deviate from the node's routine behavior. This method functions as a warning system, alerting operators to irregular log events that may signify underlying issues, thereby facilitating proactive intervention. |
| title | Unsupervised Log Anomaly Detection with Few Unique Tokens |
| topic | Cryptography and Security |
| url | https://arxiv.org/abs/2310.08951 |