Cover Image

NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Junkai, Li, Zhenhao, Hu, Xing, Xia, Xin
Format:	Preprint
Published:	2024
Subjects:	Software Engineering Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Reasoning Runtime Behavior of a Program with LLM: How Far Are We?
by: Chen, Junkai, et al.
Published: (2024)

Understanding Practitioners' Expectations on Clear Code Review Comments
by: Chen, Junkai, et al.
Published: (2024)

An Empirical Study of Speculative Decoding on Software Engineering Tasks
by: Li, Yijia, et al.
Published: (2026)

SecureVibeBench: Benchmarking Secure Vibe Coding of AI Agents via Reconstructing Vulnerability-Introducing Scenarios
by: Chen, Junkai, et al.
Published: (2025)

LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMs
by: Xia, Yunhui, et al.
Published: (2025)

Open the Oyster: Empirical Evaluation and Improvement of Code Reasoning Confidence in LLMs
by: Wang, Shufan, et al.
Published: (2025)

Enhancing Code LLMs with Reinforcement Learning in Code Generation: A Survey
by: Wang, Junqiao, et al.
Published: (2024)

An Empirical Study of Retrieval-Augmented Code Generation: Challenges and Opportunities
by: Yang, Zezhou, et al.
Published: (2025)

PerfCodeBench: Benchmarking LLMs for System-Level High-Performance Code Optimization
by: Jing, Huihao, et al.
Published: (2026)

Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEval
by: Wu, Jiarong, et al.
Published: (2025)

CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
by: Chen, Zaoyu, et al.
Published: (2026)

CodeS: Natural Language to Code Repository via Multi-Layer Sketch
by: Zan, Daoguang, et al.
Published: (2024)

Evaluation of Code LLMs on Geospatial Code Generation
by: Gramacki, Piotr, et al.
Published: (2024)

Showing LLM-Generated Code Selectively Based on Confidence of LLMs
by: Li, Jia, et al.
Published: (2024)

CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
by: Tian, Yuchen, et al.
Published: (2024)

AetherCode: Evaluating LLMs' Ability to Win In Premier Programming Competitions
by: Wang, Zihan, et al.
Published: (2025)

InstructCoder: Instruction Tuning Large Language Models for Code Editing
by: Li, Kaixin, et al.
Published: (2023)

Humanity's Last Code Exam: Can Advanced LLMs Conquer Human's Hardest Code Competition?
by: Li, Xiangyang, et al.
Published: (2025)

HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
by: Peng, Qiwei, et al.
Published: (2024)

Model Editing for LLMs4Code: How Far are We?
by: Li, Xiaopeng, et al.
Published: (2024)

LLM-as-a-Judge for Reference-less Automatic Code Validation and Refinement for Natural Language to Bash in IT Automation
by: Vo, Ngoc Phuoc An, et al.
Published: (2025)

Defects4Log: Benchmarking LLMs for Logging Code Defect Detection and Reasoning
by: Wang, Xin, et al.
Published: (2025)

Bridging Code Graphs and Large Language Models for Better Code Understanding
by: Chen, Zeqi, et al.
Published: (2025)

CodeOCR: On the Effectiveness of Vision Language Models in Code Understanding
by: Shi, Yuling, et al.
Published: (2026)

Can Language Models Replace Programmers for Coding? REPOCOD Says 'Not Yet'
by: Liang, Shanchao, et al.
Published: (2024)

OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs
by: Ahmad, Wasi Uddin, et al.
Published: (2025)

Leveraging Print Debugging to Improve Code Generation in Large Language Models
by: Hu, Xueyu, et al.
Published: (2024)

What's Wrong with Your Code Generated by Large Language Models? An Extensive Study
by: Dou, Shihan, et al.
Published: (2024)

VersiCode: Towards Version-controllable Code Generation
by: Wu, Tongtong, et al.
Published: (2024)

Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generation
by: Zhong, Li, et al.
Published: (2023)

FairCoder: Evaluating Social Bias of LLMs in Code Generation
by: Du, Yongkang, et al.
Published: (2025)

Coffee: Boost Your Code LLMs by Fixing Bugs with Feedback
by: Moon, Seungjun, et al.
Published: (2023)

CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?
by: Zhao, Yuwei, et al.
Published: (2024)

CREME: Robustness Enhancement of Code LLMs via Layer-Aware Model Editing
by: Liu, Shuhan, et al.
Published: (2025)

To Diff or Not to Diff? Structure-Aware and Adaptive Output Formats for Efficient LLM-based Code Editing
by: Cheng, Wei, et al.
Published: (2026)

Vulnerability Detection with Code Language Models: How Far Are We?
by: Ding, Yangruibo, et al.
Published: (2024)

From Output to Evaluation: Does Raw Instruction-Tuned Code LLMs Output Suffice for Fill-in-the-Middle Code Generation?
by: Ahmad, Wasi Uddin, et al.
Published: (2025)

Mercury: A Code Efficiency Benchmark for Code Large Language Models
by: Du, Mingzhe, et al.
Published: (2024)

A Semantic-based Optimization Approach for Repairing LLMs: Case Study on Code Generation
by: Gu, Jian, et al.
Published: (2025)

LongCodeZip: Compress Long Context for Code Language Models
by: Shi, Yuling, et al.
Published: (2025)