Cover Image

Improving Pretraining Data Using Perplexity Correlations

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Thrush, Tristan, Potts, Christopher, Hashimoto, Tatsunori
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Synthetic Data for any Differentiable Target
by: Thrush, Tristan, et al.
Published: (2026)

Language Models with Conformal Factuality Guarantees
by: Mohri, Christopher, et al.
Published: (2024)

Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
by: Ankner, Zachary, et al.
Published: (2024)

Understanding Finetuning for Factual Knowledge Extraction
by: Ghosal, Gaurav, et al.
Published: (2024)

Temporal Entailment Pretraining for Clinical Language Models over EHR Data
by: Tanaka, Tatsunori, et al.
Published: (2025)

Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training
by: Liu, Lei, et al.
Published: (2025)

Agentic Adversarial QA for Improving Domain-Specific LLMs
by: Grari, Vincent, et al.
Published: (2026)

Rethinking Perplexity: Revealing the Impact of Input Length on Perplexity Evaluation in LLMs
by: Cheng, Letian, et al.
Published: (2026)

Observational Scaling Laws and the Predictability of Language Model Performance
by: Ruan, Yangjun, et al.
Published: (2024)

Putting It All into Context: Simplifying Agents with LCLMs
by: Jiang, Mingjian, et al.
Published: (2025)

Linguistic Calibration of Long-Form Generations
by: Band, Neil, et al.
Published: (2024)

Robust Distortion-free Watermarks for Language Models
by: Kuditipudi, Rohith, et al.
Published: (2023)

Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with 100+ NLP Researchers
by: Si, Chenglei, et al.
Published: (2024)

The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas
by: Si, Chenglei, et al.
Published: (2025)

Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
by: Dubois, Yann, et al.
Published: (2024)

On the Learnability of Watermarks for Language Models
by: Gu, Chenchen, et al.
Published: (2023)

Reasoning to Learn from Latent Thoughts
by: Ruan, Yangjun, et al.
Published: (2025)

Embedding And Clustering Your Data Can Improve Contrastive Pretraining
by: Merrick, Luke
Published: (2024)

What is Wrong with Perplexity for Long-context Language Modeling?
by: Fang, Lizhe, et al.
Published: (2024)

Language Models Improve When Pretraining Data Matches Target Tasks
by: Mizrahi, David, et al.
Published: (2025)

Synthetic continued pretraining
by: Yang, Zitong, et al.
Published: (2024)

Graph-based Uncertainty Metrics for Long-form Language Model Outputs
by: Jiang, Mingjian, et al.
Published: (2024)

Low-Perplexity LLM-Generated Sequences and Where To Find Them
by: Wuhrmann, Arthur, et al.
Published: (2025)

AutoBencher: Towards Declarative Benchmark Construction
by: Li, Xiang Lisa, et al.
Published: (2024)

Rethinking GSPO: The Perplexity-Entropy Equivalence
by: Liu, Chi
Published: (2025)

Auditing Prompt Caching in Language Model APIs
by: Gu, Chenchen, et al.
Published: (2025)

Alzheimer's Dementia Detection Using Perplexity from Paired Large Language Models
by: Xiao, Yao, et al.
Published: (2025)

Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models
by: Klein, Tassilo, et al.
Published: (2024)

Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information
by: Hu, Zhengmian, et al.
Published: (2023)

Towards Execution-Grounded Automated AI Research
by: Si, Chenglei, et al.
Published: (2026)

Data-Centric Lessons To Improve Speech-Language Pretraining
by: Udandarao, Vishaal, et al.
Published: (2025)

Momentum Point-Perplexity Mechanics in Large Language Models
by: Tomaz, Lorenzo, et al.
Published: (2025)

Perplexity Cannot Always Tell Right from Wrong
by: Veličković, Petar, et al.
Published: (2026)

Demystifying Verbatim Memorization in Large Language Models
by: Huang, Jing, et al.
Published: (2024)

I am a Strange Dataset: Metalinguistic Tests for Language Models
by: Thrush, Tristan, et al.
Published: (2024)

pyvene: A Library for Understanding and Improving PyTorch Models via Interventions
by: Wu, Zhengxuan, et al.
Published: (2024)

Group-Level Data Selection for Efficient Pretraining
by: Yu, Zichun, et al.
Published: (2025)

A Survey on Data Selection for Language Models
by: Albalak, Alon, et al.
Published: (2024)

Inconsistent Tokenizations Cause Language Models to be Perplexed by Japanese Grammar
by: Gambardella, Andrew, et al.
Published: (2025)

Do LLMs Find Human Answers To Fact-Driven Questions Perplexing? A Case Study on Reddit
by: Seegmiller, Parker, et al.
Published: (2024)