Cover Image

Correlating and Predicting Human Evaluations of Language Models from Natural Language Processing Benchmarks

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Schaeffer, Rylan, Koura, Punit Singh, Tang, Binh, Subramanian, Ranjan, Singh, Aaditya K, Mihaylov, Todor, Bhargava, Prajjwal, Madaan, Lovish, Chatterji, Niladri S., Goswami, Vedanuj, Edunov, Sergey, Hupkes, Dieuwke, Koyejo, Sanmi, Narang, Sharan
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Quantifying Variance in Evaluation Benchmarks
by: Madaan, Lovish, et al.
Published: (2024)

Compute Optimal Scaling of Skills: Knowledge vs Reasoning
by: Roberts, Nicholas, et al.
Published: (2025)

Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models
by: Madaan, Lovish, et al.
Published: (2024)

Pretraining Scaling Laws for Generative Evaluations of Language Models
by: Schaeffer, Rylan, et al.
Published: (2025)

In-Context Learning of Energy Functions
by: Schaeffer, Rylan, et al.
Published: (2024)

Optimizing Pretraining Data Mixtures with LLM-Estimated Utility
by: Held, William, et al.
Published: (2025)

Position: Model Collapse Does Not Mean What You Think
by: Schaeffer, Rylan, et al.
Published: (2025)

Efficient Prediction of Pass@k Scaling in Large Language Models
by: Kazdan, Joshua, et al.
Published: (2025)

From Form(s) to Meaning: Probing the Semantic Depths of Language Models Using Multisense Consistency
by: Ohmer, Xenia, et al.
Published: (2024)

Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Data
by: Miranda, Brando, et al.
Published: (2023)

MultiLoKo: a multilingual local knowledge benchmark for LLMs spanning 31 languages
by: Hupkes, Dieuwke, et al.
Published: (2025)

Investigating Data Contamination for Pre-training Language Models
by: Jiang, Minhao, et al.
Published: (2024)

Understanding Adversarial Transfer: Why Representation-Space Attacks Fail Where Data-Space Attacks Succeed
by: Gupta, Isha, et al.
Published: (2025)

HARP: A challenging human-annotated math reasoning benchmark
by: Yue, Albert S., et al.
Published: (2024)

Interpretability of Language Models via Task Spaces
by: Weber, Lucas, et al.
Published: (2024)

What Causes Polysemanticity? An Alternative Origin Story of Mixed Selectivity from Incidental Causes
by: Lecomte, Victor, et al.
Published: (2023)

ZIP-FIT: Embedding-Free Data Selection via Compression-Based Alignment
by: Obbad, Elyas, et al.
Published: (2024)

Evaluating the Robustness of Chinchilla Compute-Optimal Scaling
by: Schaeffer, Rylan, et al.
Published: (2025)

Evaluation data contamination in LLMs: how do we measure it and (when) does it matter?
by: Singh, Aaditya K., et al.
Published: (2024)

How Do Large Language Monkeys Get Their Power (Laws)?
by: Schaeffer, Rylan, et al.
Published: (2025)

Beyond Verifiable Rewards: Scaling Reinforcement Learning for Language Models to Unverifiable Data
by: Tang, Yunhao, et al.
Published: (2025)

Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World
by: Kazdan, Joshua, et al.
Published: (2024)

Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness
by: Denisov-Blanch, Yegor, et al.
Published: (2026)

No, of Course I Can! Deeper Fine-Tuning Attacks That Bypass Token-Level Safety Mechanisms
by: Kazdan, Joshua, et al.
Published: (2025)

Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
by: Thakur, Aman Singh, et al.
Published: (2024)

Discovering Implicit Large Language Model Alignment Objectives
by: Chen, Edward, et al.
Published: (2026)

BTS: Harmonizing Specialized Experts into a Generalist LLM
by: Zhang, Qizhen, et al.
Published: (2025)

Scale Dependent Data Duplication
by: Kazdan, Joshua, et al.
Published: (2026)

Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?
by: Schaeffer, Rylan, et al.
Published: (2024)

LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling
by: Mondorf, Philipp, et al.
Published: (2026)

Min-p, Max Exaggeration: A Critical Analysis of Min-p Sampling in Language Models
by: Schaeffer, Rylan, et al.
Published: (2025)

Why Do Safety Guardrails Degrade Across Languages?
by: Zhang, Max, et al.
Published: (2026)

Causally Inspired Regularization Enables Domain General Representations
by: Salaudeen, Olawale, et al.
Published: (2024)

Let's Measure Information Step-by-Step: AI-Based Evaluation Beyond Vibes
by: Robertson, Zachary, et al.
Published: (2025)

CURE: Cultural Understanding and Reasoning Evaluation - A Framework for "Thick" Culture Alignment Evaluation in LLMs
by: Vo, Truong, et al.
Published: (2025)

Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs
by: Bansal, Rachit, et al.
Published: (2025)

Best-of-N Jailbreaking
by: Hughes, John, et al.
Published: (2024)

Benign Overfitting without Linearity: Neural Network Classifiers Trained by Gradient Descent for Noisy Linear Data
by: Frei, Spencer, et al.
Published: (2022)

A Framework for Objective-Driven Dynamical Stochastic Fields
by: Zhang, Yibo Jacky, et al.
Published: (2025)

Failures to Find Transferable Image Jailbreaks Between Vision-Language Models
by: Schaeffer, Rylan, et al.
Published: (2024)