Handling Large-Scale Network Flow Records: A Comparative Study on Lossy Compression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909596986114048 |
|---|---|
| author | Merlach, Gabriele Ravalico, Damiano Trevisan, Martino Palmese, Fabio Baccichet, Giovanni Redondi, Alessandro E. C. |
| author_facet | Merlach, Gabriele Ravalico, Damiano Trevisan, Martino Palmese, Fabio Baccichet, Giovanni Redondi, Alessandro E. C. |
| contents | Flow records, that summarize the characteristics of traffic flows, represent a practical and powerful way to monitor a network. While they already offer significant compression compared to full packet captures, their sheer volume remains daunting, especially for large Internet Service Providers (ISPs). In this paper, we investigate several lossy compression techniques to further reduce storage requirements while preserving the utility of flow records for key tasks, such as predicting the domain name of contacted servers. Our study evaluates scalar quantization, Principal Component Analysis (PCA), and vector quantization, applied to a real-world dataset from an operational campus network. Results reveal that scalar quantization provides the best tradeoff between compression and accuracy. PCA can preserve predictive accuracy but hampers subsequent entropic compression, and while vector quantization shows promise, it struggles with scalability due to the high-dimensional nature of the data. These findings result in practical strategies for optimizing flow record storage in large-scale monitoring scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_20729 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Handling Large-Scale Network Flow Records: A Comparative Study on Lossy Compression Merlach, Gabriele Ravalico, Damiano Trevisan, Martino Palmese, Fabio Baccichet, Giovanni Redondi, Alessandro E. C. Networking and Internet Architecture Flow records, that summarize the characteristics of traffic flows, represent a practical and powerful way to monitor a network. While they already offer significant compression compared to full packet captures, their sheer volume remains daunting, especially for large Internet Service Providers (ISPs). In this paper, we investigate several lossy compression techniques to further reduce storage requirements while preserving the utility of flow records for key tasks, such as predicting the domain name of contacted servers. Our study evaluates scalar quantization, Principal Component Analysis (PCA), and vector quantization, applied to a real-world dataset from an operational campus network. Results reveal that scalar quantization provides the best tradeoff between compression and accuracy. PCA can preserve predictive accuracy but hampers subsequent entropic compression, and while vector quantization shows promise, it struggles with scalability due to the high-dimensional nature of the data. These findings result in practical strategies for optimizing flow record storage in large-scale monitoring scenarios. |
| title | Handling Large-Scale Network Flow Records: A Comparative Study on Lossy Compression |
| topic | Networking and Internet Architecture |
| url | https://arxiv.org/abs/2504.20729 |