Cover Image

LLM-Powered Benchmark Factory: Reliable, Generic, and Efficient

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yuan, Peiwen, Feng, Shaoxiong, Li, Yiwei, Wang, Xinglin, Zhang, Yueqi, Shi, Jiayi, Tan, Chuyi, Pan, Boyuan, Hu, Yao, Li, Kan
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator
by: Yuan, Peiwen, et al.
Published: (2025)

From Sub-Ability Diagnosis to Human-Aligned Generation: Bridging the Gap for Text Length Control via MARKERGEN
by: Yuan, Peiwen, et al.
Published: (2025)

Make Every Penny Count: Difficulty-Adaptive Self-Consistency for Cost-Efficient Reasoning
by: Wang, Xinglin, et al.
Published: (2024)

UniCBE: An Uniformity-driven Comparing Based Evaluation Framework with Unified Multi-Objective Optimization
by: Yuan, Peiwen, et al.
Published: (2025)

Mind the Quote: Enabling Quotation-Aware Dialogue in LLMs via Plug-and-Play Modules
by: Zhang, Yueqi, et al.
Published: (2025)

Do Not Waste Your Rollouts: Recycling Search Experience for Efficient Test-Time Scaling
by: Wang, Xinglin, et al.
Published: (2026)

InsBank: Evolving Instruction Subset for Ongoing Alignment
by: Shi, Jiayi, et al.
Published: (2025)

Beyond One-Size-Fits-All: Tailored Benchmarks for Efficient Evaluation
by: Yuan, Peiwen, et al.
Published: (2025)

Revisiting Self-Consistency from Dynamic Distributional Alignment Perspective on Answer Aggregation
by: Li, Yiwei, et al.
Published: (2025)

Diagnosing and Mitigating System Bias in Self-Rewarding RL
by: Tan, Chuyi, et al.
Published: (2025)

Speculative Decoding for Multi-Sample Inference
by: Li, Yiwei, et al.
Published: (2025)

Focused Large Language Models are Stable Many-Shot Learners
by: Yuan, Peiwen, et al.
Published: (2024)

On Time, Within Budget: Constraint-Driven Online Resource Allocation for Agentic Workflows
by: Wang, Xinglin, et al.
Published: (2026)

Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling
by: Wang, Xinglin, et al.
Published: (2025)

Poor-Supervised Evaluation for SuperLLM via Mutual Consistency
by: Yuan, Peiwen, et al.
Published: (2024)

Instruction Embedding: Latent Representations of Instructions Towards Task Identification
by: Li, Yiwei, et al.
Published: (2024)

Integrate the Essence and Eliminate the Dross: Fine-Grained Self-Consistency for Free-Form Language Generation
by: Wang, Xinglin, et al.
Published: (2024)

Learning More from Less: Unlocking Internal Representations for Benchmark Compression
by: Zhang, Yueqi, et al.
Published: (2026)

BatchEval: Towards Human-like Text Evaluation
by: Yuan, Peiwen, et al.
Published: (2023)

CogLM: Tracking Cognitive Development of Large Language Models
by: Wang, Xinglin, et al.
Published: (2024)

PatternKV: Flattening KV Representation Expands Quantization Headroom
by: Zhang, Ji, et al.
Published: (2025)

Generative Dense Retrieval: Memory Can Be a Burden
by: Yuan, Peiwen, et al.
Published: (2024)

Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning
by: Li, Yiwei, et al.
Published: (2024)

Dynamic Stochastic Decoding Strategy for Open-Domain Dialogue Generation
by: Li, Yiwei, et al.
Published: (2024)

ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training
by: Lin, Wenxiang, et al.
Published: (2026)

Defense Against Syntactic Textual Backdoor Attacks with Token Substitution
by: Li, Xinglin, et al.
Published: (2024)

FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents
by: Li, Bobo, et al.
Published: (2025)

Bridging Internal Probability and Self-Consistency for Effective and Efficient LLM Reasoning
by: Zhou, Zhi, et al.
Published: (2025)

From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers
by: Duan, Shaoxiong, et al.
Published: (2023)

Stitch and Tell: A Structured Multimodal Data Augmentation Method for Spatial Understanding
by: Yin, Hang, et al.
Published: (2025)

Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
by: Yeh, Samuel, et al.
Published: (2025)

PMMT: Preference Alignment in Multilingual Machine Translation via LLM Distillation
by: Sun, Shuqiao, et al.
Published: (2024)

From XAI to Stories: A Factorial Study of LLM-Generated Explanation Quality
by: Lukassen, Fabian, et al.
Published: (2026)

GroupTravelBench: Benchmarking LLM Agents on Multi-Person Travel Planning
by: Cheng, Xiang, et al.
Published: (2026)

Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference
by: Yuan, Jiayi, et al.
Published: (2025)

MCiteBench: A Multimodal Benchmark for Generating Text with Citations
by: Hu, Caiyu, et al.
Published: (2025)

TurnaboutLLM: A Deductive Reasoning Benchmark from Detective Games
by: Yuan, Yuan, et al.
Published: (2025)

When Helpers Become Hazards: A Benchmark for Analyzing Multimodal LLM-Powered Safety in Daily Life
by: Lou, Xinyue, et al.
Published: (2026)

Multi-Task Reinforcement Learning for Enhanced Multimodal LLM-as-a-Judge
by: Wu, Junjie, et al.
Published: (2026)

EASYTOOL: Enhancing LLM-based Agents with Concise Tool Instruction
by: Yuan, Siyu, et al.
Published: (2024)