The Use of Large Language Models (LLM) for Cyber Threat Intelligence (CTI) in Cybercrime Forums
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916418753134592 |
|---|---|
| author | Clairoux-Trepanier, Vanessa Beauchamp, Isa-May Ruellan, Estelle Paquet-Clouston, Masarah Paquette, Serge-Olivier Clay, Eric |
| author_facet | Clairoux-Trepanier, Vanessa Beauchamp, Isa-May Ruellan, Estelle Paquet-Clouston, Masarah Paquette, Serge-Olivier Clay, Eric |
| contents | Large language models (LLMs) can be used to analyze cyber threat intelligence (CTI) data from cybercrime forums, which contain extensive information and key discussions about emerging cyber threats. However, to date, the level of accuracy and efficiency of LLMs for such critical tasks has yet to be thoroughly evaluated. Hence, this study assesses the performance of an LLM system built on the OpenAI GPT-3.5-turbo model [8] to extract CTI information. To do so, a random sample of more than 700 daily conversations from three cybercrime forums - XSS, Exploit_in, and RAMP - was extracted, and the LLM system was instructed to summarize the conversations and predict 10 key CTI variables, such as whether a large organization and/or a critical infrastructure is being targeted, with only simple human-language instructions. Then, two coders reviewed each conversation and evaluated whether the information extracted by the LLM was accurate. The LLM system performed well, with an average accuracy score of 96.23%, an average precision of 90% and an average recall of 88.2%. Various ways to enhance the model were uncovered, such as the need to help the LLM distinguish between stories and past events, as well as being careful with verb tenses in prompts. Nevertheless, the results of this study highlight the relevance of using LLMs for cyber threat intelligence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_03354 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Use of Large Language Models (LLM) for Cyber Threat Intelligence (CTI) in Cybercrime Forums Clairoux-Trepanier, Vanessa Beauchamp, Isa-May Ruellan, Estelle Paquet-Clouston, Masarah Paquette, Serge-Olivier Clay, Eric Cryptography and Security Artificial Intelligence Computation and Language Large language models (LLMs) can be used to analyze cyber threat intelligence (CTI) data from cybercrime forums, which contain extensive information and key discussions about emerging cyber threats. However, to date, the level of accuracy and efficiency of LLMs for such critical tasks has yet to be thoroughly evaluated. Hence, this study assesses the performance of an LLM system built on the OpenAI GPT-3.5-turbo model [8] to extract CTI information. To do so, a random sample of more than 700 daily conversations from three cybercrime forums - XSS, Exploit_in, and RAMP - was extracted, and the LLM system was instructed to summarize the conversations and predict 10 key CTI variables, such as whether a large organization and/or a critical infrastructure is being targeted, with only simple human-language instructions. Then, two coders reviewed each conversation and evaluated whether the information extracted by the LLM was accurate. The LLM system performed well, with an average accuracy score of 96.23%, an average precision of 90% and an average recall of 88.2%. Various ways to enhance the model were uncovered, such as the need to help the LLM distinguish between stories and past events, as well as being careful with verb tenses in prompts. Nevertheless, the results of this study highlight the relevance of using LLMs for cyber threat intelligence. |
| title | The Use of Large Language Models (LLM) for Cyber Threat Intelligence (CTI) in Cybercrime Forums |
| topic | Cryptography and Security Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2408.03354 |