Cover Image

CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Quan, Shanghaoran, Yang, Jiaxi, Yu, Bowen, Zheng, Bo, Liu, Dayiheng, Yang, An, Ren, Xuancheng, Gao, Bofei, Miao, Yibo, Feng, Yunlong, Wang, Zekun, Yang, Jian, Cui, Zeyu, Fan, Yang, Zhang, Yichang, Hui, Binyuan, Lin, Junyang
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Evaluating and Aligning CodeLLMs on Human Preference
by: Yang, Jian, et al.
Published: (2024)

Multi-Agent Collaboration for Multilingual Code Instruction Tuning
by: Yang, Jian, et al.
Published: (2025)

Aligning CodeLLMs with Direct Preference Optimization
by: Miao, Yibo, et al.
Published: (2024)

Language Models can Self-Lengthen to Generate Long Texts
by: Quan, Shanghaoran, et al.
Published: (2024)

Qwen2.5-Coder Technical Report
by: Hui, Binyuan, et al.
Published: (2024)

Turning the Tide: Repository-based Code Reflection
by: Zhang, Wei, et al.
Published: (2025)

Aggregating Elo Ratings: An Axiomatization
by: Seven, Mehmet Mars
Published: (2026)

Elo Ratings in the Presence of Intransitivity
by: Hamilton, Adam H., et al.
Published: (2024)

Evaluating and Achieving Controllable Code Completion in Code LLM
by: Zhang, Jiajun, et al.
Published: (2026)

TextClass Benchmark: A Continuous Elo Rating of LLMs in Social Sciences
by: González-Bustamante, Bastián
Published: (2024)

ExecRepoBench: Multi-level Executable Code Completion Evaluation
by: Yang, Jian, et al.
Published: (2024)

Towards Better Correctness and Efficiency in Code Generation
by: Feng, Yunlong, et al.
Published: (2025)

Empirical parameterization of the Elo Rating System
by: Maitra, Shirsa, et al.
Published: (2025)

Achieving the Highest Possible Elo Rating
by: Shah, Rikhav
Published: (2022)

Parallel Scaling Law for Language Models
by: Chen, Mouxiang, et al.
Published: (2025)

Synthesizing Text-to-SQL Data from Weak and Strong LLMs
by: Yang, Jiaxi, et al.
Published: (2024)

An Analysis of Elo Rating Systems via Markov Chains
by: Olesker-Taylor, Sam, et al.
Published: (2024)

Scaling Agentic Verifier for Competitive Coding
by: Ma, Zeyao, et al.
Published: (2026)

Keeping Elo alive: Evaluating and improving measurement properties of learning systems based on Elo ratings
by: Maria Bolsinova, et al.
Published: (2025)

Hidden Elo: Private Matchmaking through Encrypted Rating Systems
by: Budzys, Mindaugas, et al.
Published: (2026)

Is Elo Rating Reliable? A Study Under Model Misspecification
by: Tang, Shange, et al.
Published: (2025)

Elo-Evolve: A Co-evolutionary Framework for Language Model Alignment
by: Zhao, Jing, et al.
Published: (2026)

Automatically Generating Numerous Context-Driven SFT Data for LLMs across Diverse Granularity
by: Quan, Shanghaoran
Published: (2024)

New insights into Elo algorithm for practitioners and statisticians
by: Szczecinski, Leszek
Published: (2026)

Convergence and stationary distribution of Elo rating systems
by: Cortez, Roberto, et al.
Published: (2024)

Estimating abilities with an Elo-informed growth model
by: Sigfrid, Karl, et al.
Published: (2024)

SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Manner
by: Zhang, Lei, et al.
Published: (2025)

Tournament of Prompts: Evolving LLM Instructions Through Structured Debates and Elo Ratings
by: Nair, Anirudh, et al.
Published: (2025)

An Elo-based rating system for TopCoder SRM
by: Batty, Fred
Published: (2019)

Adaptation of the Multi-Concept Multivariate Elo Rating System to Medical Students Training Data
by: Kandemir, Erva Nihan, et al.
Published: (2024)

START: Self-taught Reasoner with Tools
by: Li, Chengpeng, et al.
Published: (2025)

From Completion to Editing: Unlocking Context-Aware Code Infilling via Search-and-Replace Instruction Tuning
by: Zhang, Jiajun, et al.
Published: (2026)

PlotCraft: Pushing the Limits of LLMs for Complex and Interactive Data Visualization
by: Zhang, Jiajun, et al.
Published: (2025)

Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Models
by: Gao, Bofei, et al.
Published: (2024)

Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Framework
by: Rackauckas, Zackary, et al.
Published: (2024)

DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling
by: Quan, Shanghaoran
Published: (2024)

OJBench: A Competition Level Code Benchmark For Large Language Models
by: Wang, Zhexu, et al.
Published: (2025)

Advancing Harmful Content Detection in Organizational Research: Integrating Large Language Models with Elo Rating System
by: Akben, Mustafa, et al.
Published: (2025)

Research Power Ranking: Adapting the Elo System to Quantify Scientist Evaluation
by: Knar, Eldar
Published: (2025)

Modeling LLM Agent Reviewer Dynamics in Elo-Ranked Review System
by: Huang, Hsiang-Wei, et al.
Published: (2026)