Bandit on the Hunt: Dynamic Crawling for Cyber Threat Intelligence
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915258843529216 |
|---|---|
| author | Kuehn, Philipp Nadermahmoodi, Dilara Bayer, Markus Reuter, Christian |
| author_facet | Kuehn, Philipp Nadermahmoodi, Dilara Bayer, Markus Reuter, Christian |
| contents | Public information contains valuable Cyber Threat Intelligence (CTI) that is used to prevent future attacks. While standards exist for sharing this information, much appears in non-standardized news articles or blogs. Monitoring online sources for threats is time-consuming and source selection is uncertain. Current research focuses on extracting Indicators of Compromise from known sources, rarely addressing new source identification. This paper proposes a CTI-focused crawler using multi-armed bandit (MAB) and various crawling strategies. It employs SBERT to identify relevant documents while dynamically adapting its crawling path. Our system ThreatCrawl achieves a harvest rate exceeding 25% and expands its seed by over 300% while maintaining topical focus. Additionally, the crawler identifies previously unknown but highly relevant overview pages, datasets, and domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_18375 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Bandit on the Hunt: Dynamic Crawling for Cyber Threat Intelligence Kuehn, Philipp Nadermahmoodi, Dilara Bayer, Markus Reuter, Christian Cryptography and Security Public information contains valuable Cyber Threat Intelligence (CTI) that is used to prevent future attacks. While standards exist for sharing this information, much appears in non-standardized news articles or blogs. Monitoring online sources for threats is time-consuming and source selection is uncertain. Current research focuses on extracting Indicators of Compromise from known sources, rarely addressing new source identification. This paper proposes a CTI-focused crawler using multi-armed bandit (MAB) and various crawling strategies. It employs SBERT to identify relevant documents while dynamically adapting its crawling path. Our system ThreatCrawl achieves a harvest rate exceeding 25% and expands its seed by over 300% while maintaining topical focus. Additionally, the crawler identifies previously unknown but highly relevant overview pages, datasets, and domains. |
| title | Bandit on the Hunt: Dynamic Crawling for Cyber Threat Intelligence |
| topic | Cryptography and Security |
| url | https://arxiv.org/abs/2504.18375 |