Image de couverture de livre

HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language Models on cross-domain multi-file project problems

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Xing, Jun, Bhatia, Mayur, Phulwani, Sahil, Suresh, Darshan, Matta, Rafik
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Software Engineering
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

An abstraction for solving multi-domain problems using finite element methods
par: Sagiyama, Koki, et autres
Publié: (2025)

GrowthHacker: Automated Off-Policy Evaluation Optimization Using Code-Modifying LLM Agents
par: Wu, Jie JW, et autres
Publié: (2025)

Social Media Reactions to Open Source Promotions: AI-Powered GitHub Projects on Hacker News
par: Meakpaiboonwattana, Prachnachai, et autres
Publié: (2025)

ASTRA: Autonomous Spatial-Temporal Red-teaming for AI Software Assistants
par: Xu, Xiangzhe, et autres
Publié: (2025)

MathDuels: Evaluating LLMs as Problem Posers and Solvers
par: Xu, Zhiqiu, et autres
Publié: (2026)

Hackers or Hallucinators? A Comprehensive Analysis of LLM-Based Automated Penetration Testing
par: Peng, Jiaren, et autres
Publié: (2026)

MCeT: Behavioral Model Correctness Evaluation using Large Language Models
par: Ahmed, Khaled, et autres
Publié: (2025)

CodeHacker: Automated Test Case Generation for Detecting Vulnerabilities in Competitive Programming Solutions
par: Shi, Jingwei, et autres
Publié: (2026)

Using a Feedback Loop for LLM-based Infrastructure as Code Generation
par: Palavalli, Mayur Amarnath, et autres
Publié: (2024)

Holistic Evaluation of State-of-the-Art LLMs for Code Generation
par: Zhang, Le, et autres
Publié: (2025)

SweRank+: Multilingual, Multi-Turn Code Ranking for Software Issue Localization
par: Reddy, Revanth Gangi, et autres
Publié: (2025)

A novel instance‐based method for cross‐project just‐in‐time defect prediction
par: Xiaoyan Zhu, et autres
Publié: (2024)

Component Matching Approach in Linking Business and Application Architecture
par: Kamath, Suresh
Publié: (2024)

Component Matching as a Graph Matching Problem
par: Kamath, Suresh
Publié: (2024)

Program Structure Aware Precondition Generation
par: Dinella, Elizabeth, et autres
Publié: (2023)

Conditional Execution of Transpiler Passes Based on Per-Script Feature Detection
par: Bhatia, Rishipal Singh
Publié: (2026)

Statistical Confidence in Functional Correctness: An Approach for AI Product Functional Correctness Evaluation
par: Albertini, Wallace, et autres
Publié: (2026)

SweRank: Software Issue Localization with Code Ranking
par: Reddy, Revanth Gangi, et autres
Publié: (2025)

ScalerEval: Automated and Consistent Evaluation Testbed for Auto-scalers in Microservices
par: Xie, Shuaiyu, et autres
Publié: (2025)

Assessing Coherency and Consistency of Code Execution Reasoning by Large Language Models
par: Liu, Changshu, et autres
Publié: (2025)

Assessing and Advancing Benchmarks for Evaluating Large Language Models in Software Engineering Tasks
par: Hu, Xing, et autres
Publié: (2025)

EvaluateXAI: A Framework to Evaluate the Reliability and Consistency of Rule-based XAI Techniques for Software Analytics Tasks
par: Awal, Md Abdul, et autres
Publié: (2024)

Harnessing Large Language Models for Seed Generation in Greybox Fuzzing
par: Shi, Wenxuan, et autres
Publié: (2024)

IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities
par: Li, Ziyang, et autres
Publié: (2024)

Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis
par: Deshpande, Darshan, et autres
Publié: (2026)

Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities
par: Khare, Avishree, et autres
Publié: (2023)

Testing and Evaluation of Large Language Models: Correctness, Non-Toxicity, and Fairness
par: Wang, Wenxuan
Publié: (2024)

Calibration and Correctness of Language Models for Code
par: Spiess, Claudio, et autres
Publié: (2024)

Lightweight Model Editing for LLMs to Correct Deprecated API Recommendations
par: Lin, Guancheng, et autres
Publié: (2025)

Evaluating LLMs Effectiveness in Detecting and Correcting Test Smells: An Empirical Study
par: Santana Jr, E. G., et autres
Publié: (2025)

ConAIR:Consistency-Augmented Iterative Interaction Framework to Enhance the Reliability of Code Generation
par: Dong, Jinhao, et autres
Publié: (2024)

Analyzing the Instability of Large Language Models in Automated Bug Injection and Correction
par: Er, Mehmet Bilal, et autres
Publié: (2025)

ATime-Consistent Benchmark for Repository-Level Software Engineering Evaluation
par: Xianpeng, et autres
Publié: (2026)

An Empirical Study of Self-Admitted Technical Debt in Machine Learning Software
par: Bhatia, Aaditya, et autres
Publié: (2023)

SemGuard: Real-Time Semantic Evaluator for Correcting LLM-Generated Code
par: Wang, Qinglin, et autres
Publié: (2025)

Evaluating Robustness of Large Language Models in Enterprise Applications: Benchmarks for Perturbation Consistency Across Formats and Languages
par: Bogavelli, Tara, et autres
Publié: (2026)

Towards Richer Challenge Problems for Scientific Computing Correctness
par: Sottile, Matthew, et autres
Publié: (2025)

Verify Implementation Equivalence of Large Models
par: Zhan, Qi, et autres
Publié: (2026)

Where's the Bug? Attention Probing for Scalable Fault Localization
par: Stein, Adam, et autres
Publié: (2025)

Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs
par: Li, Ziyu, et autres
Publié: (2024)