Zipf-Gramming: Scaling Byte N-Grams Up to Production Sized Malware Corpora
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918206726209536 |
|---|---|
| author | Raff, Edward Curtin, Ryan R. Everett, Derek Joyce, Robert J. Holt, James |
| author_facet | Raff, Edward Curtin, Ryan R. Everett, Derek Joyce, Robert J. Holt, James |
| contents | A classifier using byte n-grams as features is the only approach we have found fast enough to meet requirements in size (sub 2 MB), speed (multiple GB/s), and latency (sub 10 ms) for deployment in numerous malware detection scenarios. However, we've consistently found that 6-8 grams achieve the best accuracy on our production deployments but have been unable to deploy regularly updated models due to the high cost of finding the top-k most frequent n-grams over terabytes of executable programs. Because the Zipfian distribution well models the distribution of n-grams, we exploit its properties to develop a new top-k n-gram extractor that is up to $35\times$ faster than the previous best alternative. Using our new Zipf-Gramming algorithm, we are able to scale up our production training set and obtain up to 30\% improvement in AUC at detecting new malware. We show theoretically and empirically that our approach will select the top-k items with little error and the interplay between theory and engineering required to achieve these results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_13808 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Zipf-Gramming: Scaling Byte N-Grams Up to Production Sized Malware Corpora Raff, Edward Curtin, Ryan R. Everett, Derek Joyce, Robert J. Holt, James Cryptography and Security Machine Learning Mathematical Software A classifier using byte n-grams as features is the only approach we have found fast enough to meet requirements in size (sub 2 MB), speed (multiple GB/s), and latency (sub 10 ms) for deployment in numerous malware detection scenarios. However, we've consistently found that 6-8 grams achieve the best accuracy on our production deployments but have been unable to deploy regularly updated models due to the high cost of finding the top-k most frequent n-grams over terabytes of executable programs. Because the Zipfian distribution well models the distribution of n-grams, we exploit its properties to develop a new top-k n-gram extractor that is up to $35\times$ faster than the previous best alternative. Using our new Zipf-Gramming algorithm, we are able to scale up our production training set and obtain up to 30\% improvement in AUC at detecting new malware. We show theoretically and empirically that our approach will select the top-k items with little error and the interplay between theory and engineering required to achieve these results. |
| title | Zipf-Gramming: Scaling Byte N-Grams Up to Production Sized Malware Corpora |
| topic | Cryptography and Security Machine Learning Mathematical Software |
| url | https://arxiv.org/abs/2511.13808 |