Cover Image

The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakage

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Oh, Byung-Doh, Zhu, Hongao, Schuler, William
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times
by: Oh, Byung-Doh, et al.
Published: (2024)

The Impact of Token Granularity on the Predictive Power of Language Model Surprisal
by: Oh, Byung-Doh, et al.
Published: (2024)

Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilities
by: Oh, Byung-Doh, et al.
Published: (2024)

Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal
by: Nair, Sathvik, et al.
Published: (2026)

Linear Recency Bias During Training Improves Transformers' Fit to Reading Times
by: Clark, Christian, et al.
Published: (2024)

How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor?
by: Clark, Christian, et al.
Published: (2025)

Vectors from Larger Language Models Predict Human Reading Time and fMRI Data More Poorly when Dimensionality Expansion is Controlled
by: Lin, Yi-Chien, et al.
Published: (2025)

Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorly
by: Lin, Yi-Chien, et al.
Published: (2025)

Synthetic Pre-Pre-Training Improves Language Model Robustness to Noisy Pre-Training Data
by: Guo, Xu, et al.
Published: (2026)

Inside the Black Box: Detecting Data Leakage in Pre-trained Language Encoders
by: Xin, Yuan, et al.
Published: (2024)

Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokens
by: Zhang, Anqi, et al.
Published: (2024)

Sequence-Level Leakage Risk of Training Data in Large Language Models
by: Tiwari, Trishita, et al.
Published: (2024)

Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Training
by: Wang, Zhijun, et al.
Published: (2025)

Scaling Down Semantic Leakage: Investigating Associative Bias in Smaller Language Models
by: Smilga, Veronika
Published: (2025)

Tracing Privacy Leakage of Language Models to Training Data via Adjusted Influence Functions
by: Liu, Jinxin, et al.
Published: (2024)

Improving Language Models Trained on Translated Data with Continual Pre-Training and Dictionary Learning Analysis
by: Boughorbel, Sabri, et al.
Published: (2024)

Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMs
by: Oh, Gyutaek, et al.
Published: (2025)

The Frequency Confound in Language-Model Surprisal and Metaphor Novelty
by: Momen, Omar, et al.
Published: (2026)

Verifiable by Design: Aligning Language Models to Quote from Pre-Training Data
by: Zhang, Jingyu, et al.
Published: (2024)

Breaking Language Barriers: Cross-Lingual Continual Pre-Training at Scale
by: Zheng, Wenzhen, et al.
Published: (2024)

An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisal
by: Yoshida, Ryo, et al.
Published: (2026)

Simulating Training Data Leakage in Multiple-Choice Benchmarks for LLM Evaluation
by: Hidayat, Naila Shafirni, et al.
Published: (2025)

Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
by: Gallifant, Jack, et al.
Published: (2024)

LLMs for automatic annotation of Mandarin narrative transcripts
by: Zhao, Qingwen, et al.
Published: (2026)

D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Models
by: Que, Haoran, et al.
Published: (2024)

On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
by: Zhang, Charlie, et al.
Published: (2025)

FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language
by: Penedo, Guilherme, et al.
Published: (2025)

ISACL: Internal State Analyzer for Copyrighted Training Data Leakage
by: Zhang, Guangwei, et al.
Published: (2025)

Exploring Forgetting in Large Language Model Pre-Training
by: Liao, Chonghua, et al.
Published: (2024)

Quantifying Memorization and Detecting Training Data of Pre-trained Language Models using Japanese Newspaper
by: Ishihara, Shotaro, et al.
Published: (2024)

Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data
by: Kondo, Minato, et al.
Published: (2024)

AutoScale: Scale-Aware Data Mixing for Pre-Training LLMs
by: Kang, Feiyang, et al.
Published: (2024)

Confabulation: The Surprising Value of Large Language Model Hallucinations
by: Sui, Peiqi, et al.
Published: (2024)

Testing the Predictions of Surprisal Theory in 11 Languages
by: Wilcox, Ethan Gotlieb, et al.
Published: (2023)

"According to ...": Prompting Language Models Improves Quoting from Pre-Training Data
by: Weller, Orion, et al.
Published: (2023)

Language Model Behavioral Phases are Consistent Across Architecture, Training Data, and Scale
by: Michaelov, James A., et al.
Published: (2025)

A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese
by: Liu, Yikang, et al.
Published: (2024)

Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training
by: Jiang, Changhao, et al.
Published: (2025)

Pitfalls of Scale: Investigating the Inverse Task of Redefinition in Large Language Models
by: Stringli, Elena, et al.
Published: (2025)

Evolution of Concepts in Language Model Pre-Training
by: Ge, Xuyang, et al.
Published: (2025)