Copertina

Maintaining MTEB: Towards Long Term Usability and Reproducibility of Embedding Benchmarks

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Chung, Isaac, Kerboua, Imene, Kardos, Marton, Solomatin, Roman, Enevoldsen, Kenneth
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Computation and Language Artificial Intelligence Software Engineering
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

MIEB: Massive Image Embedding Benchmark
di: Xiao, Chenghao, et al.
Pubblicazione: (2025)

The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding
di: Enevoldsen, Kenneth, et al.
Pubblicazione: (2024)

YABLoCo: Yet Another Benchmark for Long Context Code Generation
di: Valeev, Aidar, et al.
Pubblicazione: (2025)

ConvCodeWorld: Benchmarking Conversational Code Generation in Reproducible Feedback Environments
di: Han, Hojae, et al.
Pubblicazione: (2025)

Rigor, Reliability, and Reproducibility Matter: A Decade-Scale Survey of 572 Code Benchmarks
di: Cao, Jialun, et al.
Pubblicazione: (2025)

SWE-CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration
di: Chen, Jialong, et al.
Pubblicazione: (2026)

Can Coding Agents Reproduce Findings in Computational Materials Science?
di: Huang, Ziyang, et al.
Pubblicazione: (2026)

Towards Automated Smart Contract Generation: Evaluation, Benchmarking, and Retrieval-Augmented Repair
di: Chen, Zaoyu, et al.
Pubblicazione: (2025)

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
di: Orlanski, Gabriel, et al.
Pubblicazione: (2026)

OmniCode: A Benchmark for Evaluating Software Engineering Agents
di: Sonwane, Atharv, et al.
Pubblicazione: (2026)

MTEB-French: Resources for French Sentence Embedding Evaluation and Analysis
di: Ciancone, Mathieu, et al.
Pubblicazione: (2024)

BenchGuard: Who Guards the Benchmarks? Automated Auditing of LLM Agent Benchmarks
di: Tu, Xinming, et al.
Pubblicazione: (2026)

IndustryCode: A Benchmark for Industry Code Generation
di: Zeng, Puyu, et al.
Pubblicazione: (2026)

BenchBrowser: Retrieving Evidence for Evaluating Benchmark Validity
di: Diddee, Harshita, et al.
Pubblicazione: (2026)

Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
di: Zan, Daoguang, et al.
Pubblicazione: (2025)

A Code Comprehension Benchmark for Large Language Models for Code
di: Havare, Jayant, et al.
Pubblicazione: (2025)

StructEval: Benchmarking LLMs' Capabilities to Generate Structural Outputs
di: Yang, Jialin, et al.
Pubblicazione: (2025)

CodeJudgeBench: Benchmarking LLM-as-a-Judge for Coding Tasks
di: Jiang, Hongchao, et al.
Pubblicazione: (2025)

Benchmarking Open-Source Safety Guard Models: A Comprehensive Evaluation
di: Harsh, Reetu Raj, et al.
Pubblicazione: (2026)

ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development
di: Yang, Jie, et al.
Pubblicazione: (2026)

AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms
di: Zhao, Haoyu, et al.
Pubblicazione: (2026)

Collu-Bench: A Benchmark for Predicting Language Model Hallucinations in Code
di: Jiang, Nan, et al.
Pubblicazione: (2024)

SpecBench: Measuring Reward Hacking in Long-Horizon Coding Agents
di: Zhao, Bingchen, et al.
Pubblicazione: (2026)

SWE-bench-java: A GitHub Issue Resolving Benchmark for Java
di: Zan, Daoguang, et al.
Pubblicazione: (2024)

SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgrades
di: Lam, Man Ho, et al.
Pubblicazione: (2026)

Beyond Correctness: Benchmarking Multi-dimensional Code Generation for Large Language Models
di: Zheng, Jiasheng, et al.
Pubblicazione: (2024)

Towards an Understanding of Context Utilization in Code Intelligence
di: Wang, Yanlin, et al.
Pubblicazione: (2025)

CangjieBench: Benchmarking LLMs on a Low-Resource General-Purpose Programming Language
di: Cheng, Junhang, et al.
Pubblicazione: (2026)

BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions
di: Zhuo, Terry Yue, et al.
Pubblicazione: (2024)

Dynamic Benchmarking of Reasoning Capabilities in Code Large Language Models Under Data Contamination
di: Chen, Simin, et al.
Pubblicazione: (2025)

The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligence
di: White, Matt, et al.
Pubblicazione: (2024)

E2Edev: Benchmarking Large Language Models in End-to-End Software Development Task
di: Liu, Jingyao, et al.
Pubblicazione: (2025)

UA-Code-Bench: A Competitive Programming Benchmark for Evaluating LLM Code Generation in Ukrainian
di: Syromiatnikov, Mykyta, et al.
Pubblicazione: (2025)

SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents
di: Zhang, Zhirui, et al.
Pubblicazione: (2026)

RepoTransBench: A Real-World Multilingual Benchmark for Repository-Level Code Translation
di: Wang, Yanli, et al.
Pubblicazione: (2024)

EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories
di: Li, Jia, et al.
Pubblicazione: (2024)

SWE-QA-Pro: A Representative Benchmark and Scalable Training Recipe for Repository-Level Code Understanding
di: Cai, Songcheng, et al.
Pubblicazione: (2026)

SWE-MERA: A Dynamic Benchmark for Agenticly Evaluating Large Language Models on Software Engineering Tasks
di: Adamenko, Pavel, et al.
Pubblicazione: (2025)

CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation
di: Yan, Weixiang, et al.
Pubblicazione: (2023)

MoSE: Hierarchical Self-Distillation Enhances Early Layer Embeddings
di: Gurioli, Andrea, et al.
Pubblicazione: (2025)