Cover Image

What is Wrong with Perplexity for Long-context Language Modeling?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Fang, Lizhe, Wang, Yifei, Liu, Zhaoyang, Zhang, Chenheng, Jegelka, Stefanie, Gao, Jinyang, Ding, Bolin, Wang, Yisen
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

A Theoretical Understanding of Self-Correction through In-context Alignment
by: Wang, Yifei, et al.
Published: (2024)

When More is Less: Understanding Chain-of-Thought Length in LLMs
by: Wu, Yuyang, et al.
Published: (2025)

Long-Short Alignment for Effective Long-Context Modeling in LLMs
by: Du, Tianqi, et al.
Published: (2025)

Advancing LLM Safe Alignment with Safety Representation Ranking
by: Du, Tianqi, et al.
Published: (2025)

Autoregressive Models Rival Diffusion Models at ANY-ORDER Generation
by: Du, Tianqi, et al.
Published: (2026)

LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs
by: Li, Ang, et al.
Published: (2025)

How to Craft Backdoors with Unlabeled Data Alone?
by: Wang, Yifei, et al.
Published: (2024)

Rethinking Invariance in In-context Learning
by: Fang, Lizhe, et al.
Published: (2025)

A Canonicalization Perspective on Invariant and Equivariant Learning
by: Ma, George, et al.
Published: (2024)

Perplexity Cannot Always Tell Right from Wrong
by: Veličković, Petar, et al.
Published: (2026)

G1: Teaching LLMs to Reason on Graphs with Reinforcement Learning
by: Guo, Xiaojun, et al.
Published: (2025)

Rethinking Perplexity: Revealing the Impact of Input Length on Perplexity Evaluation in LLMs
by: Cheng, Letian, et al.
Published: (2026)

Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
by: Ankner, Zachary, et al.
Published: (2024)

Beyond Interpretability: The Gains of Feature Monosemanticity on Model Robustness
by: Zhang, Qi, et al.
Published: (2024)

Look Ahead or Look Around? A Theoretical Comparison Between Autoregressive and Masked Pretraining
by: Zhang, Qi, et al.
Published: (2024)

Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations
by: Wei, Zeming, et al.
Published: (2023)

Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training
by: Liu, Lei, et al.
Published: (2025)

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization
by: Wu, Junkang, et al.
Published: (2024)

Rethinking Personalization in Large Language Models at the Token Level
by: Zhang, Chenheng, et al.
Published: (2026)

Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models
by: Klein, Tassilo, et al.
Published: (2024)

Output Scaling: YingLong-Delayed Chain of Thought in a Large Pretrained Time Series Forecasting Model
by: Wang, Xue, et al.
Published: (2025)

Momentum Point-Perplexity Mechanics in Large Language Models
by: Tomaz, Lorenzo, et al.
Published: (2025)

Understanding the Role of Equivariance in Self-supervised Learning
by: Wang, Yifei, et al.
Published: (2024)

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization
by: Wu, Junkang, et al.
Published: (2024)

LongReward: Improving Long-context Large Language Models with AI Feedback
by: Zhang, Jiajie, et al.
Published: (2024)

Inconsistent Tokenizations Cause Language Models to be Perplexed by Japanese Grammar
by: Gambardella, Andrew, et al.
Published: (2025)

Stepwise Perplexity-Guided Refinement for Efficient Chain-of-Thought Reasoning in Large Language Models
by: Cui, Yingqian, et al.
Published: (2025)

Can In-context Learning Really Generalize to Out-of-distribution Tasks?
by: Wang, Qixun, et al.
Published: (2024)

On the Emergence of Position Bias in Transformers
by: Wu, Xinyi, et al.
Published: (2025)

Rethinking GSPO: The Perplexity-Entropy Equivalence
by: Liu, Chi
Published: (2025)

Larger or Smaller Reward Margins to Select Preferences for Alignment?
by: Huang, Kexin, et al.
Published: (2025)

Curse of High Dimensionality Issue in Transformer for Long-context Modeling
by: Zhang, Shuhai, et al.
Published: (2025)

Language Ranker: A Lightweight Ranking framework for LLM Decoding
by: Zhang, Chenheng, et al.
Published: (2025)

Can Perplexity Predict Fine-tuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali
by: Luitel, Nishant, et al.
Published: (2024)

Alzheimer's Dementia Detection Using Perplexity from Paired Large Language Models
by: Xiao, Yao, et al.
Published: (2025)

A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms
by: Gong, Ruihao, et al.
Published: (2024)

Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models
by: Qiu, Yifu, et al.
Published: (2025)

Improving Pretraining Data Using Perplexity Correlations
by: Thrush, Tristan, et al.
Published: (2024)

Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
by: Chen, Yanxi, et al.
Published: (2024)

Activation Control for Efficiently Eliciting Long Chain-of-thought Ability of Language Models
by: Zhao, Zekai, et al.
Published: (2025)