Cover Image

Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Heineman, David, Hofmann, Valentin, Magnusson, Ian, Gu, Yuling, Smith, Noah A., Hajishirzi, Hannaneh, Lo, Kyle, Dodge, Jesse
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Fluid Language Model Benchmarking
by: Hofmann, Valentin, et al.
Published: (2025)

OLMES: A Standard for Language Model Evaluations
by: Gu, Yuling, et al.
Published: (2024)

Paloma: A Benchmark for Evaluating Language Model Fit
by: Magnusson, Ian, et al.
Published: (2023)

Merge to Learn: Efficiently Adding Skills to Language Models with Model Merging
by: Morrison, Jacob, et al.
Published: (2024)

Olmix: A Framework for Data Mixing Throughout LM Development
by: Chen, Mayee F., et al.
Published: (2026)

Establishing Task Scaling Laws via Compute-Efficient Model Ladders
by: Bhagia, Akshita, et al.
Published: (2024)

Set the Clock: Temporal Alignment of Pretrained Language Models
by: Zhao, Bowen, et al.
Published: (2024)

ComPO: Community Preferences for Language Model Personalization
by: Kumar, Sachin, et al.
Published: (2024)

HREF: Human Response-Guided Evaluation of Instruction Following in Language Models
by: Lyu, Xinxi, et al.
Published: (2024)

APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inference
by: Zhao, Bowen, et al.
Published: (2024)

RewardBench 2: Advancing Reward Model Evaluation
by: Malik, Saumya, et al.
Published: (2025)

Organize the Web: Constructing Domains Enhances Pre-Training Data Curation
by: Wettig, Alexander, et al.
Published: (2025)

Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index
by: Xu, Hao, et al.
Published: (2025)

ScienceMeter: Tracking Scientific Knowledge Updates in Language Models
by: Wang, Yike, et al.
Published: (2025)

BTR: Binary Token Representations for Efficient Retrieval Augmented Language Models
by: Cao, Qingqing, et al.
Published: (2023)

SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
by: Min, Sewon, et al.
Published: (2023)

DataDecide: How to Predict Best Pretraining Data with Small Experiments
by: Magnusson, Ian, et al.
Published: (2025)

Scalable Data Ablation Approximations for Language Models through Modular Training and Merging
by: Na, Clara, et al.
Published: (2024)

EvalTree: Profiling Language Model Weaknesses via Hierarchical Capability Trees
by: Zeng, Zhiyuan, et al.
Published: (2025)

What's In My Big Data?
by: Elazar, Yanai, et al.
Published: (2023)

Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
by: Soldaini, Luca, et al.
Published: (2024)

TurnWise: The Gap between Single- and Multi-turn Language Model Capabilities
by: Graf, Victoria, et al.
Published: (2026)

Husky: A Unified, Open-Source Language Agent for Multi-Step Reasoning
by: Kim, Joongwon, et al.
Published: (2024)

Steering off Course: Reliability Challenges in Steering Language Models
by: Da Silva, Patrick Queiroz, et al.
Published: (2025)

OLMo: Accelerating the Science of Language Models
by: Groeneveld, Dirk, et al.
Published: (2024)

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
by: Liu, Jiacheng, et al.
Published: (2024)

ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data
by: Chen, Tong, et al.
Published: (2025)

ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Context
by: Kim, Joongwon, et al.
Published: (2025)

Fine-grained Hallucination Detection and Editing for Language Models
by: Mishra, Abhika, et al.
Published: (2024)

Investigating Human-Aligned Large Language Model Uncertainty
by: Moore, Kyle, et al.
Published: (2025)

Learning to Detect Language Model Training Data via Active Reconstruction
by: Yin, Junjie Oscar, et al.
Published: (2026)

Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models
by: Moore, Kyle, et al.
Published: (2025)

Data Engineering for Scaling Language Models to 128K Context
by: Fu, Yao, et al.
Published: (2024)

Language Models Hallucinate, but May Excel at Fact Verification
by: Guan, Jian, et al.
Published: (2023)

OLMoE: Open Mixture-of-Experts Language Models
by: Muennighoff, Niklas, et al.
Published: (2024)

Small Reward Models via Backward Inference
by: Wang, Yike, et al.
Published: (2026)

MentorCollab: Selective Large-to-Small Inference-Time Guidance for Efficient Reasoning
by: Wang, Haojin, et al.
Published: (2026)

Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
by: Ivison, Hamish, et al.
Published: (2024)

SciRIFF: A Resource to Enhance Language Model Instruction-Following over Scientific Literature
by: Wadden, David, et al.
Published: (2024)

Answer, Assemble, Ace: Understanding How LMs Answer Multiple Choice Questions
by: Wiegreffe, Sarah, et al.
Published: (2024)