Cover Image

Training on the Test Task Confounds Evaluation and Emergence

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Dominguez-Olmedo, Ricardo, Dorner, Florian E., Hardt, Moritz
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Train-before-Test Harmonizes Language Model Rankings
by: Zhang, Guanhua, et al.
Published: (2025)

Lawma: The Power of Specialization for Legal Annotation
by: Dominguez-Olmedo, Ricardo, et al.
Published: (2024)

Answer Matching Outperforms Multiple Choice for Language Model Evaluation
by: Chandak, Nikhil, et al.
Published: (2025)

Computational Arbitrage in AI Model Markets
by: Olmedo, Ricardo, et al.
Published: (2026)

Test-Time Training on Nearest Neighbors for Large Language Models
by: Hardt, Moritz, et al.
Published: (2023)

FutureSim: Replaying World Events to Evaluate Adaptive Agents
by: Goel, Shashwat, et al.
Published: (2026)

Whose Preferences? Differences in Fairness Preferences and Their Impact on the Fairness of AI Utilizing Human Feedback
by: Lerner, Emilia Agis, et al.
Published: (2024)

Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domains
by: Xu, Austin, et al.
Published: (2025)

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
by: Han, Seungwook, et al.
Published: (2024)

Probing the Emergence of Cross-lingual Alignment during LLM Training
by: Wang, Hetong, et al.
Published: (2024)

In-Place Test-Time Training
by: Feng, Guhao, et al.
Published: (2026)

Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences
by: Ghoshal, Asish, et al.
Published: (2022)

Self-Trained Verification for Training- and Test-Time Self-Improvement
by: Wu, Chen Henry, et al.
Published: (2026)

Don't Label Twice: Quantity Beats Quality when Comparing Binary Classifiers on a Budget
by: Dorner, Florian E., et al.
Published: (2024)

XL-Suite: Cross-Lingual Synthetic Training and Evaluation Data for Open-Ended Generation
by: Iyer, Vivek, et al.
Published: (2025)

The Unreasonable Effectiveness of Easy Training Data for Hard Tasks
by: Hase, Peter, et al.
Published: (2024)

Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training
by: Mistry, Deven Mahesh, et al.
Published: (2025)

Test-Time Detoxification without Training or Learning Anything
by: Saglam, Baturay, et al.
Published: (2026)

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning
by: Akyürek, Ekin, et al.
Published: (2024)

SR-TTT: Surprisal-Aware Residual Test-Time Training
by: P, Swamynathan V
Published: (2026)

Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack
by: Xu, Xiaoyue, et al.
Published: (2024)

Test-Time Scaling in Reasoning Models Is Not Effective for Knowledge-Intensive Tasks Yet
by: Zhao, James Xu, et al.
Published: (2025)

Evaluating Language-Model Agents on Realistic Autonomous Tasks
by: Kinniment, Megan, et al.
Published: (2023)

Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards
by: Padula, Alexander G., et al.
Published: (2024)

Query-Conditioned Test-Time Self-Training for Large Language Models
by: Song, Chaehee, et al.
Published: (2026)

MesaNet: Sequence Modeling by Locally Optimal Test-Time Training
by: von Oswald, Johannes, et al.
Published: (2025)

A Dataset for Evaluating LLM-based Evaluation Functions for Research Question Extraction Task
by: Fujisaki, Yuya, et al.
Published: (2024)

Task-Stratified Knowledge Scaling Laws for Post-Training Quantized Large Language Models
by: Zhou, Chenxi, et al.
Published: (2025)

Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task
by: Li, Kenneth, et al.
Published: (2022)

CityBench: Evaluating the Capabilities of Large Language Models for Urban Tasks
by: Feng, Jie, et al.
Published: (2024)

Visualizing Uncertainty in Translation Tasks: An Evaluation of LLM Performance and Confidence Metrics
by: Park, Jin Hyun, et al.
Published: (2025)

DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
by: Zhu, Kaijie, et al.
Published: (2023)

Enhancing LLM Reasoning via Critique Models with Test-Time and Training-Time Supervision
by: Xi, Zhiheng, et al.
Published: (2024)

Becoming Experienced Judges: Selective Test-Time Learning for Evaluators
by: Jwa, Seungyeon, et al.
Published: (2025)

Learning from Relevant Subgoals in Successful Dialogs using Iterative Training for Task-oriented Dialog Systems
by: Kaiser, Magdalena, et al.
Published: (2024)

Challenging the Validity of Personality Tests for Large Language Models
by: Sühr, Tom, et al.
Published: (2023)

Training and Evaluating Language Models with Template-based Data Generation
by: Zhang, Yifan
Published: (2024)

Culinary Class Wars: Evaluating LLMs using ASH in Cuisine Transfer Task
by: Lee, Hoonick, et al.
Published: (2024)

TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scaling
by: Lin, Weizhe, et al.
Published: (2025)

Universal Response and Emergence of Induction in LLMs
by: Luick, Niclas
Published: (2024)