Cover Image

ScratchEval : A Multimodal Evaluation Framework for LLMs in Block-Based Programming

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Si, Yuan, Han, Simeng, Li, Daming, Shi, Hanyuan, Zhang, Jialu
Format:	Preprint
Published:	2026
Subjects:	Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

EcoScratch: Cost-Effective Multimodal Repair for Scratch Using Execution Feedback
by: Si, Yuan, et al.
Published: (2026)

ViScratch: Using Large Language Models and Gameplay Videos for Automated Feedback in Scratch
by: Si, Yuan, et al.
Published: (2025)

Raven: Rethinking Automated Assessment for Scratch Programs via Video-Grounded Evaluation
by: Li, Donglin, et al.
Published: (2026)

Stitch: Step-by-step LLM Guided Tutoring for Scratch
by: Si, Yuan, et al.
Published: (2025)

A Systematic Study of Time Limit Exceeded Errors in Online Programming Assignments
by: Zhang, Jialu, et al.
Published: (2025)

Enhancing NeuroEvolution-Based Game Testing: A Branch Coverage Approach for Scratch Programs
by: Sohail, Khizra, et al.
Published: (2025)

RepoZero: Can LLMs Generate a Code Repository from Scratch?
by: Zhang, Zhaoxi, et al.
Published: (2026)

A Block-Based Testing Framework for Scratch
by: Feldmeier, Patric, et al.
Published: (2024)

SpecEval: Evaluating Code Comprehension in Large Language Models via Program Specifications
by: Ma, Lezhi, et al.
Published: (2024)

ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructions
by: Li, Jialin, et al.
Published: (2026)

SESR-Eval: Dataset for Evaluating LLMs in the Title-Abstract Screening of Systematic Reviews
by: Huotala, Aleksi, et al.
Published: (2025)

SR-Eval: Evaluating LLMs on Code Generation under Stepwise Requirement Refinement
by: Zhan, Zexun, et al.
Published: (2025)

ArkEval: Benchmarking and Evaluating Automated CodeRepair for ArkTS
by: Xie, Bang, et al.
Published: (2026)

ProgramBench: Can Language Models Rebuild Programs From Scratch?
by: Yang, John, et al.
Published: (2026)

TAM-Eval: Evaluating LLMs for Automated Unit Test Maintenance
by: Bruches, Elena, et al.
Published: (2026)

RePurr: Automated Repair of Block-Based Learners' Programs
by: Schweikl, Sebastian, et al.
Published: (2025)

LitterBox+: An Extensible Framework for LLM-enhanced Scratch Static Code Analysis
by: Fein, Benedikt, et al.
Published: (2025)

Evaluating the Generalizability of LLMs in Automated Program Repair
by: Li, Fengjie, et al.
Published: (2025)

ProjectEval: A Benchmark for Programming Agents Automated Evaluation on Project-Level Code Generation
by: Liu, Kaiyuan, et al.
Published: (2025)

TypeEvalPy: A Micro-benchmarking Framework for Python Type Inference Tools
by: Venkatesh, Ashwin Prasad Shivarpatna, et al.
Published: (2023)

ClassEval-T: Evaluating Large Language Models in Class-Level Code Translation
by: Xue, Pengyu, et al.
Published: (2024)

ScalerEval: Automated and Consistent Evaluation Testbed for Auto-scalers in Microservices
by: Xie, Shuaiyu, et al.
Published: (2025)

VulnRepairEval: An Exploit-Based Evaluation Framework for Assessing Large Language Model Vulnerability Repair Capabilities
by: Wang, Weizhe, et al.
Published: (2025)

Voice-Controlled Scratch for Children with (Motor) Disabilities
by: Goller, Elias, et al.
Published: (2026)

DependEval: Benchmarking LLMs for Repository Dependency Understanding
by: Du, Junjia, et al.
Published: (2025)

Codellm-Devkit: A Framework for Contextualizing Code LLMs with Program Analysis Insights
by: Krishna, Rahul, et al.
Published: (2024)

ScenEval: A Benchmark for Scenario-Based Evaluation of Code Generation
by: Paul, Debalina Ghosh, et al.
Published: (2024)

Can LLMs Recover Program Semantics? A Systematic Evaluation with Symbolic Execution
by: Feng, Rong, et al.
Published: (2025)

SecVulEval: Benchmarking LLMs for Real-World C/C++ Vulnerability Detection
by: Ahmed, Md Basim Uddin, et al.
Published: (2025)

HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent
by: Wu, Jie JW, et al.
Published: (2024)

VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generation
by: Vijayaraghavan, Prashanth, et al.
Published: (2024)

FeedbackEval: A Benchmark for Evaluating Large Language Models in Feedback-Driven Code Repair Tasks
by: Dai, Dekun, et al.
Published: (2025)

SolContractEval: A Benchmark for Evaluating Contract-Level Solidity Code Generation
by: Ye, Zhifan, et al.
Published: (2025)

ReleaseEval: A Benchmark for Evaluating Language Models in Automated Release Note Generation
by: Meng, Qianru, et al.
Published: (2025)

Experimenting a New Programming Practice with LLMs
by: Zhang, Simiao, et al.
Published: (2024)

SmartLLMs Scheduler: A Framework for Cost-Effective LLMs Utilization
by: Liu, Yueyue, et al.
Published: (2025)

AgentArcEval: An Architecture Evaluation Method for Foundation Model based Agents
by: Lu, Qinghua, et al.
Published: (2025)

ComplexCodeEval: A Benchmark for Evaluating Large Code Models on More Complex Code
by: Feng, Jia, et al.
Published: (2024)

LiCoEval: Evaluating LLMs on License Compliance in Code Generation
by: Xu, Weiwei, et al.
Published: (2024)

Does Teaming-Up LLMs Improve Secure Code Generation? A Comprehensive Evaluation with Multi-LLMSecCodeEval
by: Sabir, Bushra, et al.
Published: (2026)