Adaptive and Efficient Log Parsing as a Cloud Service
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916686496530432 |
|---|---|
| author | Li, Zeyan Song, Jie Zhang, Tieying Yang, Tao Ou, Xiongjun Ye, Yingjie Duan, Pengfei Lin, Muchen Chen, Jianjun |
| author_facet | Li, Zeyan Song, Jie Zhang, Tieying Yang, Tao Ou, Xiongjun Ye, Yingjie Duan, Pengfei Lin, Muchen Chen, Jianjun |
| contents | Logs are a critical data source for cloud systems, enabling advanced features like monitoring, alerting, and root cause analysis. However, the massive scale and diverse formats of unstructured logs pose challenges for adaptable, efficient, and accurate parsing methods. This paper introduces ByteBrain-LogParser, an innovative log parsing framework designed specifically for cloud environments. ByteBrain-LogParser employs a hierarchical clustering algorithm to allow real-time precision adjustments, coupled with optimizations such as positional similarity distance, deduplication, and hash encoding to enhance performance. Experiments on large-scale datasets show that it processes 229,000 logs per second on average, achieving an 840% speedup over the fastest baseline while maintaining accuracy comparable to state-of-the-art methods. Real-world evaluations further validate its efficiency and adaptability, demonstrating its potential as a robust cloud-based log parsing solution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_09113 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adaptive and Efficient Log Parsing as a Cloud Service Li, Zeyan Song, Jie Zhang, Tieying Yang, Tao Ou, Xiongjun Ye, Yingjie Duan, Pengfei Lin, Muchen Chen, Jianjun Software Engineering Logs are a critical data source for cloud systems, enabling advanced features like monitoring, alerting, and root cause analysis. However, the massive scale and diverse formats of unstructured logs pose challenges for adaptable, efficient, and accurate parsing methods. This paper introduces ByteBrain-LogParser, an innovative log parsing framework designed specifically for cloud environments. ByteBrain-LogParser employs a hierarchical clustering algorithm to allow real-time precision adjustments, coupled with optimizations such as positional similarity distance, deduplication, and hash encoding to enhance performance. Experiments on large-scale datasets show that it processes 229,000 logs per second on average, achieving an 840% speedup over the fastest baseline while maintaining accuracy comparable to state-of-the-art methods. Real-world evaluations further validate its efficiency and adaptability, demonstrating its potential as a robust cloud-based log parsing solution. |
| title | Adaptive and Efficient Log Parsing as a Cloud Service |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2504.09113 |