Image de couverture de livre

Execution-Based Evaluation of Natural Language to Bash and PowerShell for Incident Remediation

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Vo, Ngoc Phuoc An, Paulovicks, Brent, Sheinin, Vadim
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Software Engineering
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

LLM-as-a-Judge for Reference-less Automatic Code Validation and Refinement for Natural Language to Bash in IT Automation
par: Vo, Ngoc Phuoc An, et autres
Publié: (2025)

The Power of Words: Generating PowerShell Attacks from Natural Language
par: Liguori, Pietro, et autres
Publié: (2024)

PowerPeeler: A Precise and General Dynamic Deobfuscation Method for PowerShell Scripts
par: Li, Ruijie, et autres
Publié: (2024)

Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
par: Pysklo, Hubert M., et autres
Publié: (2026)

Large Language Model Critics for Execution-Free Evaluation of Code Changes
par: Yadavally, Aashish, et autres
Publié: (2025)

From SWE-ZERO to SWE-HERO: Execution-free to Execution-based Fine-tuning for Software Engineering Agents
par: Ludwig, Nikolai, et autres
Publié: (2026)

Evaluating Retrieval-Augmented Generation Variants for Natural Language-Based SQL and API Call Generation
par: Marketsmüller, Michael, et autres
Publié: (2026)

Automated Root Causing of Cloud Incidents using In-Context Learning with GPT-4
par: Zhang, Xuchao, et autres
Publié: (2024)

NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variations
par: Chen, Junkai, et autres
Publié: (2024)

GenX: Mastering Code and Test Generation with Execution Feedback
par: Wang, Nan, et autres
Publié: (2024)

Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis
par: Huang, Dong, et autres
Publié: (2025)

CodeSpecBench: Benchmarking LLMs for Executable Behavioral Specification Generation
par: Chen, Zaoyu, et autres
Publié: (2026)

CodeHalu: Investigating Code Hallucinations in LLMs via Execution-based Verification
par: Tian, Yuchen, et autres
Publié: (2024)

CodeBenchGen: Creating Scalable Execution-based Code Generation Benchmarks
par: Xie, Yiqing, et autres
Publié: (2024)

DuET: Dual Execution for Test Output Prediction with Generated Code and Pseudocode
par: Han, Hojae, et autres
Publié: (2026)

Explainable Compliance Detection with Multi-Hop Natural Language Inference on Assurance Case Structure
par: Ikhwantri, Fariz, et autres
Publié: (2025)

Talk Less, Verify More: Improving LLM Assistants with Semantic Checks and Execution Feedback
par: Sun, Yan, et autres
Publié: (2026)

SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories
par: Bogin, Ben, et autres
Publié: (2024)

StepCodeReasoner: Aligning Code Reasoning with Stepwise Execution Traces via Reinforcement Learning
par: Wang, Hao, et autres
Publié: (2026)

NExT: Teaching Large Language Models to Reason about Code Execution
par: Ni, Ansong, et autres
Publié: (2024)

Investigating Execution-Aware Language Models for Code Optimization
par: Di Menna, Federico, et autres
Publié: (2025)

AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitor
par: Yang, Shu, et autres
Publié: (2026)

The Stability Trap: Evaluating the Reliability of LLM-Based Instruction Adherence Auditing
par: Shergadwala, Murtuza N.
Publié: (2026)

English Please: Evaluating Machine Translation with Large Language Models for Multilingual Bug Reports
par: Patil, Avinash, et autres
Publié: (2025)

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
par: Yan, Weixiang, et autres
Publié: (2023)

Nissist: An Incident Mitigation Copilot based on Troubleshooting Guides
par: An, Kaikai, et autres
Publié: (2024)

HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalization
par: Peng, Qiwei, et autres
Publié: (2024)

CRITICTOOL: Evaluating Self-Critique Capabilities of Large Language Models in Tool-Calling Error Scenarios
par: Huang, Shiting, et autres
Publié: (2025)

HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation
par: Yu, Zhaojian, et autres
Publié: (2024)

Evaluate-and-Purify: Fortifying Code Language Models Against Adversarial Attacks Using LLM-as-a-Judge
par: Mu, Wenhan, et autres
Publié: (2025)

Generating and Evaluating Sustainable Procurement Criteria for the Swiss Public Sector using In-Context Prompting with Large Language Models
par: Gao, Yingqiang, et autres
Publié: (2026)

SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysis
par: Li, Yansong, et autres
Publié: (2025)

Is Your AI-Generated Code Really Safe? Evaluating Large Language Models on Secure Code Generation with CodeSecEval
par: Wang, Jiexin, et autres
Publié: (2024)

Debug like a Human: A Large Language Model Debugger via Verifying Runtime Execution Step-by-step
par: Zhong, Li, et autres
Publié: (2024)

Divide-and-Conquer Meets Consensus: Unleashing the Power of Functions in Code Generation
par: Chen, Jingchang, et autres
Publié: (2024)

FaultProfIT: Hierarchical Fault Profiling of Incident Tickets in Large-scale Cloud Systems
par: Huang, Junjie, et autres
Publié: (2024)

OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
par: Zheng, Tianyu, et autres
Publié: (2024)

CodeS: Natural Language to Code Repository via Multi-Layer Sketch
par: Zan, Daoguang, et autres
Publié: (2024)

CPSLint: A Domain-Specific Language Providing Data Validation and Sanitisation for Industrial Cyber-Physical Systems
par: Odyurt, Uraz, et autres
Publié: (2025)

KAIJU: An Executive Kernel for Intent-Gated Execution of LLM Agents
par: Guerin, Cormac, et autres
Publié: (2026)