Cover Image

LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ullah, Saad, Han, Mingji, Pujar, Saurabh, Pearce, Hammond, Coskun, Ayse, Stringhini, Gianluca
Format:	Preprint
Published:	2023
Subjects:	Cryptography and Security
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

From CVE Entries to Verifiable Exploits: An Automated Multi-Agent Framework for Reproducing CVEs
by: Ullah, Saad, et al.
Published: (2025)

LLMs Cannot Reliably Judge (Yet?): A Comprehensive Assessment on the Robustness of LLM-as-a-Judge
by: Li, Songze, et al.
Published: (2025)

LOKI: Proactively Discovering Online Scam Websites by Mining Toxic Search Queries
by: Paudel, Pujan, et al.
Published: (2025)

SecureFalcon: Are We There Yet in Automated Software Vulnerability Detection with LLMs?
by: Ferrag, Mohamed Amine, et al.
Published: (2023)

Logic Meets Magic: LLMs Cracking Smart Contract Vulnerabilities
by: Xiao, ZeKe, et al.
Published: (2025)

LASHED: LLMs And Static Hardware Analysis for Early Detection of RTL Bugs
by: Ahmad, Baleegh, et al.
Published: (2025)

Large Language Models Cannot Reliably Detect Vulnerabilities in JavaScript: The First Systematic Benchmark and Evaluation
by: Fei, Qingyuan, et al.
Published: (2025)

Fixing Hardware Security Bugs with Large Language Models
by: Ahmad, Baleegh, et al.
Published: (2023)

The Cost of Convenience: Identifying, Analyzing, and Mitigating Predatory Loan Applications on Android
by: Akanji, Olawale Amos, et al.
Published: (2026)

Lightweight Yet Secure: Secure Scripting Language Generation via Lightweight LLMs
by: Zhang, Keyang, et al.
Published: (2026)

Adversarial Perturbations Cannot Reliably Protect Artists From Generative AI
by: Hönig, Robert, et al.
Published: (2024)

Enabling Contextual Soft Moderation on Social Media through Contrastive Textual Deviation
by: Paudel, Pujan, et al.
Published: (2024)

VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specifications
by: Zhu, Hao, et al.
Published: (2025)

Blockchain based Decentralized Petition System
by: Kaur, Jagdeep, et al.
Published: (2024)

Silent Consent, Persistent Risk: Android Permission Groups and Custom Permissions
by: Akanji, Olawale Amos, et al.
Published: (2026)

Persistent Human Feedback, LLMs, and Static Analyzers for Secure Code Generation and Vulnerability Detection
by: Firouzi, Ehsan, et al.
Published: (2026)

Benchmarking LLMs and LLM-based Agents in Practical Vulnerability Detection for Code Repositories
by: Yildiz, Alperen, et al.
Published: (2025)

On the Effectiveness of Instruction-Tuning Local LLMs for Identifying Software Vulnerabilities
by: Park, Sangryu, et al.
Published: (2025)

LLM Agents for Automated Web Vulnerability Reproduction: Are We There Yet?
by: Liu, Bin, et al.
Published: (2025)

SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Code
by: Li, Xinghang, et al.
Published: (2025)

SoK: The Security-Safety Continuum of Multimodal Foundation Models through Information Flow and Global Game-Theoretic Analysis of Asymmetric Threats
by: Sun, Ruoxi, et al.
Published: (2024)

CIBER: A Comprehensive Benchmark for Security Evaluation of Code Interpreter Agents
by: Ba, Lei, et al.
Published: (2026)

(Security) Assertions by Large Language Models
by: Kande, Rahul, et al.
Published: (2023)

Deep Learning Under Siege: Identifying Security Vulnerabilities and Risk Mitigation Strategies
by: Al-Karaki, Jamal, et al.
Published: (2024)

CS-Eval: A Comprehensive Large Language Model Benchmark for CyberSecurity
by: Yu, Zhengmin, et al.
Published: (2024)

Advancing Biomedical Signal Security: Real-Time ECG Monitoring with Chaotic Encryption
by: Yuksel, Beyazit Bestami, et al.
Published: (2024)

Security in the Era of Perceptive Networks: A Comprehensive Taxonomic Framework for Integrated Sensing and Communication Security
by: Thapa, Chandra, et al.
Published: (2026)

TUBERAIDER: Attributing Coordinated Hate Attacks on YouTube Videos to their Source Communities
by: Saeed, Mohammad Hammas, et al.
Published: (2023)

Hardware Design and Security Needs Attention: From Survey to Path Forward
by: Ghimire, Sujan, et al.
Published: (2025)

LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights
by: Sheng, Ze, et al.
Published: (2025)

Quantifying Memory Cells Vulnerability for DRAM Security
by: Hu, Zilong, et al.
Published: (2026)

Enhancing Security Testing Software for Systems that Cannot be Subjected to the Risks of Penetration Testing Through the Incorporation of Multi-threading and and Other Capabilities
by: Tassava, Matthew, et al.
Published: (2024)

Multi-Agent Taint Specification Extraction for Vulnerability Detection
by: Ghebremichael, Jonah, et al.
Published: (2026)

Position: Certified Robustness Does Not (Yet) Imply Model Security
by: Cullen, Andrew C., et al.
Published: (2025)

On the Vulnerability of Underwater Magnetic Induction Communication
by: Muzzammil, Muhammad, et al.
Published: (2025)

Security Vulnerabilities in Software Supply Chain for Autonomous Vehicles
by: Haque, Md Wasiul, et al.
Published: (2025)

A Systematization of Security Vulnerabilities in Computer Use Agents
by: Jones, Daniel, et al.
Published: (2025)

Comprehensive Vulnerability Analysis is Necessary for Trustworthy LLM-MAS
by: He, Pengfei, et al.
Published: (2025)

Everything You Wanted to Know About LLM-based Vulnerability Detection But Were Afraid to Ask
by: Li, Yue, et al.
Published: (2025)

Agentic AI for Autonomous Defense in Software Supply Chain Security: Beyond Provenance to Vulnerability Mitigation
by: Syed, Toqeer Ali, et al.
Published: (2025)