Cover Image

Characterizing and Evaluating the Reliability of LLMs against Jailbreak Attacks

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Kexin, Liu, Yi, Wang, Dongxia, Chen, Jiaying, Wang, Wenhai
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

ORFuzz: Fuzzing the "Other Side" of LLM Safety -- Testing Over-Refusal
by: Zhang, Haonan, et al.
Published: (2025)

CrossPL: Evaluating Large Language Models on Cross Programming Language Code Generation
by: Xiong, Zhanhang, et al.
Published: (2025)

Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study
by: Liu, Yi, et al.
Published: (2023)

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
by: Yan, Weixiang, et al.
Published: (2023)

Can LLMs Generate Reliable Test Case Generators? A Study on Competition-Level Programming Problems
by: Cao, Yuhan, et al.
Published: (2025)

Evaluating LLMs on Sequential API Call Through Automated Test Generation
by: Huang, Yuheng, et al.
Published: (2025)

Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities
by: Wang, Hanbin, et al.
Published: (2025)

BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluation
by: Lai, Peng, et al.
Published: (2026)

Rigor, Reliability, and Reproducibility Matter: A Decade-Scale Survey of 572 Code Benchmarks
by: Cao, Jialun, et al.
Published: (2025)

StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs
by: Yang, Jialin, et al.
Published: (2025)

Prompt Injection attack against LLM-integrated Applications
by: Liu, Yi, et al.
Published: (2023)

BigCodeArena: Unveiling More Reliable Human Preferences in Code Generation via Execution
by: Zhuo, Terry Yue, et al.
Published: (2025)

Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation
by: Zhong, Li, et al.
Published: (2023)

BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
by: Tu, Xinming, et al.
Published: (2026)

Revisiting the Reliability of Language Models in Instruction-Following
by: Dong, Jianshuo, et al.
Published: (2025)

InverseCoder: Self-improving Instruction-Tuned Code LLMs with Inverse-Instruct
by: Wu, Yutong, et al.
Published: (2024)

GameDevBench: Evaluating Agentic Capabilities Through Game Development
by: Chi, Wayne, et al.
Published: (2026)

Conjecture and Inquiry: Quantifying Software Performance Requirements via Interactive Retrieval-Augmented Preference Elicitation
by: Wang, Shihai, et al.
Published: (2026)

VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code Generation
by: Ni, Yuansheng, et al.
Published: (2025)

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
by: Chen, Jialong, et al.
Published: (2026)

CodeMind: Evaluating Large Language Models for Code Reasoning
by: Liu, Changshu, et al.
Published: (2024)

Testing and Evaluation of Large Language Models: Correctness, Non-Toxicity, and Fairness
by: Wang, Wenxuan
Published: (2024)

Exploring ChatGPT's Capabilities on Vulnerability Management
by: Liu, Peiyu, et al.
Published: (2023)

To See is Not to Master: Teaching LLMs to Use Private Libraries for Code Generation
by: Zhang, Yitong, et al.
Published: (2026)

EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repair
by: Chen, Jiaao, et al.
Published: (2026)

From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future
by: Jin, Haolin, et al.
Published: (2024)

FeatBench: Towards More Realistic Evaluation of Feature-level Code Generation
by: Chen, Haorui, et al.
Published: (2025)

Evaluation of LLMs on Syntax-Aware Code Fill-in-the-Middle Tasks
by: Gong, Linyuan, et al.
Published: (2024)

Towards Automated Smart Contract Generation: Evaluation, Benchmarking, and Retrieval-Augmented Repair
by: Chen, Zaoyu, et al.
Published: (2025)

DevEval: Evaluating Code Generation in Practical Software Projects
by: Li, Jia, et al.
Published: (2024)

Sticking to the Mean: Detecting Sticky Tokens in Text Embedding Models
by: Chen, Kexin, et al.
Published: (2025)

`Do as I say not as I do': A Semi-Automated Approach for Jailbreak Prompt Attack against Multimodal LLMs
by: Chiu, Chun Wai, et al.
Published: (2025)

LeDex: Training LLMs to Better Self-Debug and Explain Code
by: Jiang, Nan, et al.
Published: (2024)

LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigation
by: Zhang, Ziyao, et al.
Published: (2024)

Evaluation of General Large Language Models in Contextually Assessing Semantic Concepts Extracted from Adult Critical Care Electronic Health Record Notes
by: Liu, Darren, et al.
Published: (2024)

ACECODER: Acing Coder RL via Automated Test-Case Synthesis
by: Zeng, Huaye, et al.
Published: (2025)

OmniCode: A Benchmark for Evaluating Software Engineering Agents
by: Sonwane, Atharv, et al.
Published: (2026)

RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation
by: Wang, Yanli, et al.
Published: (2024)

CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
by: Cheng, Junhang, et al.
Published: (2026)

AutoCode: LLMs as Problem Setters for Competitive Programming
by: Zhou, Shang, et al.
Published: (2025)