Cover Image

LLM-as-a-Judge for Reference-less Automatic Code Validation and Refinement for Natural Language to Bash in IT Automation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Vo, Ngoc Phuoc An, Paulovicks, Brent, Sheinin, Vadim
Format:	Preprint
Published:	2025
Subjects:	Software Engineering Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Execution-Based Evaluation of Natural Language to Bash and PowerShell for Incident Remediation
by: Vo, Ngoc Phuoc An, et al.
Published: (2024)

CodeSift: An LLM-Based Reference-Less Framework for Automatic Code Validation
by: Aggarwal, Pooja, et al.
Published: (2024)

Don't Judge Code by Its Cover: Exploring Biases in LLM Judges for Code Evaluation
by: Moon, Jiwon, et al.
Published: (2025)

Rethinking Code Refinement: Learning to Judge Code Efficiency
by: Seo, Minju, et al.
Published: (2024)

CodeJudge-Eval: Can Large Language Models be Good Judges in Code Understanding?
by: Zhao, Yuwei, et al.
Published: (2024)

CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
by: Jiang, Hongchao, et al.
Published: (2025)

Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judge
by: Mu, Wenhan, et al.
Published: (2025)

CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences
by: Weyssow, Martin, et al.
Published: (2024)

BanglaForge: LLM Collaboration with Self-Refinement for Bangla Code Generation
by: Dihan, Mahir Labib, et al.
Published: (2025)

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
by: Chou, Jason, et al.
Published: (2025)

Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation
by: Fu, Lingyue, et al.
Published: (2025)

CodeJudge: Evaluating Code Generation with Large Language Models
by: Tong, Weixi, et al.
Published: (2024)

BiasScope: Towards Automated Detection of Bias in LLM-as-a-Judge Evaluation
by: Lai, Peng, et al.
Published: (2026)

CYCLE: Learning to Self-Refine the Code Generation
by: Ding, Yangruibo, et al.
Published: (2024)

NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations
by: Chen, Junkai, et al.
Published: (2024)

Enhanced Automated Code Vulnerability Repair using Large Language Models
by: de-Fitero-Dominguez, David, et al.
Published: (2024)

AutoIOT: LLM-Driven Automated Natural Language Programming for AIoT Applications
by: Shen, Leming, et al.
Published: (2025)

Self-reflection in Automated Qualitative Coding: Improving Text Annotation through Secondary LLM Critique
by: Dunivin, Zackary Okun, et al.
Published: (2026)

Iterative Refinement of Project-Level Code Context for Precise Code Generation with Compiler Feedback
by: Bi, Zhangqian, et al.
Published: (2024)

EffiBench: Benchmarking the Efficiency of Automatically Generated Code
by: Huang, Dong, et al.
Published: (2024)

ScriptSmith: A Unified LLM Framework for Enhancing IT Operations via Automated Bash Script Generation, Assessment, and Refinement
by: Chatterjee, Oishik, et al.
Published: (2024)

ProbeLLM: Automating Principled Diagnosis of LLM Failures
by: Huang, Yue, et al.
Published: (2026)

Evaluating and Achieving Controllable Code Completion in Code LLM
by: Zhang, Jiajun, et al.
Published: (2026)

Code Fingerprints: Disentangled Attribution of LLM-Generated Code
by: Guo, Jiaxun, et al.
Published: (2026)

RethinkMCTS: Refining Erroneous Thoughts in Monte Carlo Tree Search for Code Generation
by: Li, Qingyao, et al.
Published: (2024)

Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
by: Lin, Jiahang, et al.
Published: (2026)

CodeS: Natural Language to Code Repository via Multi-Layer Sketch
by: Zan, Daoguang, et al.
Published: (2024)

HarnessLLM: Automatic Testing Harness Generation via Reinforcement Learning
by: Liu, Yujian, et al.
Published: (2025)

Comparing Developer and LLM Biases in Code Evaluation
by: Mittal, Aditya, et al.
Published: (2026)

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
by: Zheng, Tianyu, et al.
Published: (2024)

Advancing Automated In-Isolation Validation in Repository-Level Code Translation
by: Ke, Kaiyao, et al.
Published: (2025)

EffiSkill: Agent Skill Based Automated Code Efficiency Optimization
by: Wang, Zimu, et al.
Published: (2026)

HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
by: Peng, Qiwei, et al.
Published: (2024)

Crystal: Illuminating LLM Abilities on Language and Code
by: Tao, Tianhua, et al.
Published: (2024)

CodeOCR: On the Effectiveness of Vision Language Models in Code Understanding
by: Shi, Yuling, et al.
Published: (2026)

LLMSniffer: Detecting LLM-Generated Code via GraphCodeBERT and Supervised Contrastive Learning
by: Dihan, Mahir Labib, et al.
Published: (2026)

From Prediction to Application: Language Model-based Code Knowledge Tracing with Domain Adaptive Pre-Training and Automatic Feedback System with Pedagogical Prompting for Comprehensive Programming Education
by: Lee, Unggi, et al.
Published: (2024)

LongCodeZip: Compress Long Context for Code Language Models
by: Shi, Yuling, et al.
Published: (2025)

Bridging Code Graphs and Large Language Models for Better Code Understanding
by: Chen, Zeqi, et al.
Published: (2025)

Mercury: A Code Efficiency Benchmark for Code Large Language Models
by: Du, Mingzhe, et al.
Published: (2024)