Cover Image

PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yadav, Ankit, Beniwal, Himanshu, Singh, Mayank
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

COMI-LINGUA: Expert Annotated Large-Scale Dataset for Multitask NLP in Hindi-English Code-Mixing
by: Sheth, Rajvee, et al.
Published: (2025)

Where Does Toxicity Live? Mechanistic Localization and Targeted Suppression in Language Models
by: Beniwal, Himanshu, et al.
Published: (2026)

Char-mander Use mBackdoor! A Study of Cross-lingual Backdoor Attacks in Multilingual LLMs
by: Beniwal, Himanshu, et al.
Published: (2025)

Cross-lingual Editing in Multilingual Language Models
by: Beniwal, Himanshu, et al.
Published: (2024)

COMMENTATOR: A Code-mixed Multilingual Text Annotation Framework
by: Sheth, Rajvee, et al.
Published: (2024)

Remember This Event That Year? Assessing Temporal Information and Reasoning in Large Language Models
by: Beniwal, Himanshu, et al.
Published: (2024)

One Instruction Does Not Fit All: How Well Do Embeddings Align Personas and Instructions in Low-Resource Indian Languages?
by: Shah, Arya, et al.
Published: (2026)

DEPART: DEcomposing PARiTy across Multilingual LLMs
by: Uppadhyay, Manan, et al.
Published: (2026)

UNITYAI-GUARD: Pioneering Toxicity Detection Across Low-Resource Indian Languages
by: Beniwal, Himanshu, et al.
Published: (2025)

Benchmarking LLMs for Pairwise Causal Discovery in Biomedical and Multi-Domain Contexts
by: Anuyah, Sydney, et al.
Published: (2026)

Breaking mBad! Supervised Fine-tuning for Cross-Lingual Detoxification
by: Beniwal, Himanshu, et al.
Published: (2025)

GRASP: A Grid-Based Benchmark for Evaluating Commonsense Spatial Reasoning
by: Tang, Zhisheng, et al.
Published: (2024)

Error Taxonomy-Guided Prompt Optimization
by: Singh, Mayank, et al.
Published: (2026)

TeXpert: A Multi-Level Benchmark for Evaluating LaTeX Code Generation by LLMs
by: Kale, Sahil, et al.
Published: (2025)

VisCoder: Fine-Tuning LLMs for Executable Python Visualization Code Generation
by: Ni, Yuansheng, et al.
Published: (2025)

NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
by: Basu, Kinjal, et al.
Published: (2024)

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
by: Yan, Weixiang, et al.
Published: (2023)

SagaScale: A Realistic, Scalable, and High-Quality Long-Context Benchmark Built from Full-Length Novels
by: Du, Guancheng, et al.
Published: (2025)

Beyond Monolingual Assumptions: A Survey of Code-Switched NLP in the Era of Large Language Models across Modalities
by: Sheth, Rajvee, et al.
Published: (2025)

DynaCode: A Dynamic Complexity-Aware Code Benchmark for Evaluating Large Language Models in Code Generation
by: Hu, Wenhao, et al.
Published: (2025)

Can GPT Redefine Medical Understanding? Evaluating GPT on Biomedical Machine Reading Comprehension
by: Vatsal, Shubham, et al.
Published: (2024)

Enhancing Presentation Slide Generation by LLMs with a Multi-Staged End-to-End Approach
by: Bandyopadhyay, Sambaran, et al.
Published: (2024)

ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Code
by: Hua, Tianyu, et al.
Published: (2025)

Redefining "Hallucination" in LLMs: Towards a psychology-informed framework for mitigating misinformation
by: Berberette, Elijah, et al.
Published: (2024)

Understanding the Role of LLMs in Multimodal Evaluation Benchmarks
by: Jiang, Botian, et al.
Published: (2024)

DHP Benchmark: Are LLMs Good NLG Evaluators?
by: Wang, Yicheng, et al.
Published: (2024)

On Robustness and Reliability of Benchmark-Based Evaluation of LLMs
by: Lunardi, Riccardo, et al.
Published: (2025)

Code-Vision: Evaluating Multimodal LLMs Logic Understanding and Code Generation Capabilities
by: Wang, Hanbin, et al.
Published: (2025)

Defining and Evaluating Decision and Composite Risk in Language Models Applied to Natural Language Inference
by: Shen, Ke, et al.
Published: (2024)

LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
by: Rando, Stefano, et al.
Published: (2025)

Task--Specificity Score: Measuring How Much Instructions Really Matter for Supervision
by: Kadasi, Pritam, et al.
Published: (2026)

Conditioning LLMs to Generate Code-Switched Text
by: Heredia, Maite, et al.
Published: (2025)

Evaluating the Generalization Ability of Quantized LLMs: Benchmark, Analysis, and Toolbox
by: Liu, Yijun, et al.
Published: (2024)

Beyond Benchmark: LLMs Evaluation with an Anthropomorphic and Value-oriented Roadmap
by: Wang, Jun, et al.
Published: (2025)

UA-Code-Bench: A Competitive Programming Benchmark for Evaluating LLM Code Generation in Ukrainian
by: Syromiatnikov, Mykyta, et al.
Published: (2025)

Benchmarking the Detection of LLMs-Generated Modern Chinese Poetry
by: Wang, Shanshan, et al.
Published: (2025)

Enhancing Cache-Augmented Generation (CAG) with Adaptive Contextual Compression for Scalable Knowledge Integration
by: Agrawal, Rishabh, et al.
Published: (2025)

DA-Code: Agent Data Science Code Generation Benchmark for Large Language Models
by: Huang, Yiming, et al.
Published: (2024)

Disce aut Deficere: Evaluating LLMs Proficiency on the INVALSI Italian Benchmark
by: Mercorio, Fabio, et al.
Published: (2024)

MM-Eval: A Hierarchical Benchmark for Modern Mongolian Evaluation in LLMs
by: Zhang, Mengyuan, et al.
Published: (2024)