Learned, Lagged, LLM-splained: LLM Responses to End User Security Questions
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911235578003456 |
|---|---|
| author | Prakash, Vijay Lee, Kevin Bhattacharya, Arkaprabha Huang, Danny Yuxing Staddon, Jessica |
| author_facet | Prakash, Vijay Lee, Kevin Bhattacharya, Arkaprabha Huang, Danny Yuxing Staddon, Jessica |
| contents | Answering end user security questions is challenging. While large language models (LLMs) like GPT, LLAMA, and Gemini are far from error-free, they have shown promise in answering a variety of questions outside of security. We studied LLM performance in the area of end user security by qualitatively evaluating 3 popular LLMs on 900 systematically collected end user security questions.
While LLMs demonstrate broad generalist ``knowledge'' of end user security information, there are patterns of errors and limitations across LLMs consisting of stale and inaccurate answers, and indirect or unresponsive communication styles, all of which impacts the quality of information received. Based on these patterns, we suggest directions for model improvement and recommend user strategies for interacting with LLMs when seeking assistance with security. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_14571 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learned, Lagged, LLM-splained: LLM Responses to End User Security Questions Prakash, Vijay Lee, Kevin Bhattacharya, Arkaprabha Huang, Danny Yuxing Staddon, Jessica Cryptography and Security Artificial Intelligence Computation and Language Human-Computer Interaction Answering end user security questions is challenging. While large language models (LLMs) like GPT, LLAMA, and Gemini are far from error-free, they have shown promise in answering a variety of questions outside of security. We studied LLM performance in the area of end user security by qualitatively evaluating 3 popular LLMs on 900 systematically collected end user security questions. While LLMs demonstrate broad generalist ``knowledge'' of end user security information, there are patterns of errors and limitations across LLMs consisting of stale and inaccurate answers, and indirect or unresponsive communication styles, all of which impacts the quality of information received. Based on these patterns, we suggest directions for model improvement and recommend user strategies for interacting with LLMs when seeking assistance with security. |
| title | Learned, Lagged, LLM-splained: LLM Responses to End User Security Questions |
| topic | Cryptography and Security Artificial Intelligence Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2411.14571 |