Cover Image

Scaling Evaluation-time Compute with Reasoning Models as Evaluators

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kim, Seungone, Wu, Ian, Lee, Jinu, Yue, Xiang, Lee, Seongyun, Moon, Mingyeong, Lawrence, Carolin, Gashteovski, Kiril, Hockenmaier, Julia, Neubig, Graham, Welleck, Sean
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Evaluating Language Models as Synthetic Data Generators
by: Kim, Seungone, et al.
Published: (2024)

Evaluating Step-by-step Reasoning Traces: A Survey
by: Lee, Jinu, et al.
Published: (2025)

The CoT Encyclopedia: Analyzing, Predicting, and Controlling how a Reasoning Model will Think
by: Lee, Seongyun, et al.
Published: (2025)

RefineBench: Evaluating Refinement Capability of Language Models via Checklists
by: Lee, Young-Jun, et al.
Published: (2025)

Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics
by: Lee, Jinu, et al.
Published: (2025)

Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models
by: Kim, Seungone, et al.
Published: (2024)

ReasoningFlow: Semantic Structure of Complex Reasoning Traces
by: Lee, Jinu, et al.
Published: (2025)

Prometheus-Vision: Vision-Language Model as a Judge for Fine-Grained Evaluation
by: Lee, Seongyun, et al.
Published: (2024)

Gym-Anything: Turn any Software into an Agent Environment
by: Aggarwal, Pranjal, et al.
Published: (2026)

Compositional Steering of Large Language Models with Steering Tokens
by: Radevski, Gorjan, et al.
Published: (2026)

Aligning to Thousands of Preferences via System Message Generalization
by: Lee, Seongyun, et al.
Published: (2024)

Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions
by: Liu, Emmy, et al.
Published: (2025)

OptimalThinkingBench: Evaluating Over and Underthinking in LLMs
by: Aggarwal, Pranjal, et al.
Published: (2025)

MEDDxAgent: A Unified Modular Agent Framework for Explainable Automatic Differential Diagnosis
by: Rose, Daniel, et al.
Published: (2025)

Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning
by: Huan, Maggie, et al.
Published: (2025)

AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
by: Gioacchini, Luca, et al.
Published: (2024)

TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Action
by: Zhou, Chenyue, et al.
Published: (2025)

Better Instruction-Following Through Minimum Bayes Risk
by: Wu, Ian, et al.
Published: (2024)

Evaluating and Designing Sparse Autoencoders by Approximating Quasi-Orthogonality
by: Lee, Sewoong, et al.
Published: (2025)

On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
by: Zhang, Charlie, et al.
Published: (2025)

VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge
by: Song, Yueqi, et al.
Published: (2025)

Verus-SpecGym: An Agentic Environment for Evaluating Specification Autoformalization
by: Agarwal, Anmol, et al.
Published: (2026)

BEOL‐Compatible Liquid‐Metal‐Printing of Ultrathin 2D Oxide Memtransistors and Its Applications in Neuromorphic Computing
by: Sanghyun Moon, et al.
Published: (2026)

Grid Spatial Understanding: A Dataset for Textual Spatial Reasoning over Grids, Embodied Settings, and Coordinate Structures
by: Sidhu, Risham, et al.
Published: (2026)

L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning
by: Aggarwal, Pranjal, et al.
Published: (2025)

Programming with Pixels: Can Computer-Use Agents do Software Engineering?
by: Aggarwal, Pranjal, et al.
Published: (2025)

Leveraging Open Information Extraction for More Robust Domain Transfer of Event Trigger Detection
by: Dukić, David, et al.
Published: (2023)

Entailment-Preserving First-order Logic Representations in Natural Language Entailment
by: Lee, Jinu, et al.
Published: (2025)

Can Language Models Evaluate Human Written Text? Case Study on Korean Student Writing for Education
by: Kim, Seungyoon, et al.
Published: (2024)

From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Models
by: Welleck, Sean, et al.
Published: (2024)

SymBa: Symbolic Backward Chaining for Structured Natural Language Reasoning
by: Lee, Jinu, et al.
Published: (2024)

The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models
by: Kim, Seungone, et al.
Published: (2024)

FREESON: Retriever-Free Retrieval-Augmented Reasoning via Corpus-Traversing MCTS
by: Kim, Chaeeun, et al.
Published: (2025)

Reasoning with Latent Tokens in Diffusion Language Models
by: He, Andre, et al.
Published: (2026)

Demystifying Long Chain-of-Thought Reasoning in LLMs
by: Yeo, Edward, et al.
Published: (2025)

Lost in the Noise: How Reasoning Models Fail with Contextual Distractors
by: Lee, Seongyun, et al.
Published: (2026)

Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities
by: Bertsch, Amanda, et al.
Published: (2025)

Bridging the Missing-Modality Gap: Improving Text-Only Calibration of Vision Language Models
by: Kim, Mingyeong, et al.
Published: (2026)

Reasoning over mathematical objects: on-policy reward modeling and test time aggregation
by: Aggarwal, Pranjal, et al.
Published: (2026)

LightPAL: Lightweight Passage Retrieval for Open Domain Multi-Document Summarization
by: Enomoto, Masafumi, et al.
Published: (2024)