Cover Image

Intrinsic Entropy of Context Length Scaling in LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Shi, Jingzhe, Ma, Qinwei, Liu, Hongyi, Zhao, Hang, Hwang, Jeng-Neng, Li, Lei
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Scaling Law for Time Series Forecasting
by: Shi, Jingzhe, et al.
Published: (2024)

Gradient Imbalance in Direct Preference Optimization
by: Ma, Qinwei, et al.
Published: (2025)

Entropy Centroids as Intrinsic Rewards for Test-Time Scaling
by: Zhao, Wenshuo, et al.
Published: (2026)

CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMs
by: Shi, Jingzhe, et al.
Published: (2024)

UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inference
by: Zhou, Lang, et al.
Published: (2026)

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
by: Ma, Xuezhe, et al.
Published: (2024)

Position: The Inevitable End of One-Architecture-Fits-All-Domains in Time Series Forecasting
by: Ma, Qinwei, et al.
Published: (2026)

Demystifying OPD: Length Inflation and Stabilization Strategies for Large Language Models
by: Luo, Feng, et al.
Published: (2026)

LANCET: Neural Intervention via Structural Entropy for Mitigating Faithfulness Hallucinations in LLMs
by: Wang, Chenxu, et al.
Published: (2026)

The Role of Deductive and Inductive Reasoning in Large Language Models
by: Cai, Chengkun, et al.
Published: (2024)

FinGPT: Enhancing Sentiment-Based Stock Movement Prediction with Dissemination-Aware and Context-Enriched LLMs
by: Liang, Yixuan, et al.
Published: (2024)

LongAct: Harnessing Intrinsic Activation Patterns for Long-Context Reinforcement Learning
by: Ping, Bowen, et al.
Published: (2026)

SELF: Self-Extend the Context Length With Logistic Growth Function
by: Dang, Phat Thanh, et al.
Published: (2025)

Improving Variable-Length Generation in Diffusion Language Models via Length Regularization
by: Cheng, Zicong, et al.
Published: (2026)

Does Alignment Tuning Really Break LLMs' Internal Confidence?
by: Oh, Hongseok, et al.
Published: (2024)

Rethinking Perplexity: Revealing the Impact of Input Length on Perplexity Evaluation in LLMs
by: Cheng, Letian, et al.
Published: (2026)

Diffusion LMs Can Approximate Optimal Infilling Lengths Implicitly
by: Liu, Hengchang, et al.
Published: (2026)

What Scales in Cross-Entropy Scaling Law?
by: Yan, Junxi, et al.
Published: (2025)

LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs
by: Bai, Yushi, et al.
Published: (2024)

Sculptor: Empowering LLMs with Cognitive Agency via Active Context Management
by: Li, Mo, et al.
Published: (2025)

Flux Attention: Context-Aware Hybrid Attention for Efficient LLMs Inference
by: Qiu, Quantong, et al.
Published: (2026)

Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models
by: Wang, Xindi, et al.
Published: (2024)

A Controlled Study on Long Context Extension and Generalization in LLMs
by: Lu, Yi, et al.
Published: (2024)

Scaling Efficient LLMs
by: Kausik, B. N.
Published: (2024)

Training-Inference Consistent Segmented Execution for Long-Context LLMs
by: Shang, Xianpeng, et al.
Published: (2026)

ALIEN: Aligned Entropy Head for Improving Uncertainty Estimation of LLMs
by: Zabolotnyi, Artem, et al.
Published: (2025)

From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers
by: Duan, Shaoxiong, et al.
Published: (2023)

Reward Is Enough: LLMs Are In-Context Reinforcement Learners
by: Song, Kefan, et al.
Published: (2025)

TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
by: Wu, Wei, et al.
Published: (2024)

Towards Self-Robust LLMs: Intrinsic Prompt Noise Resistance via CoIPO
by: Yang, Xin, et al.
Published: (2026)

Explore-Execute Chain: Towards an Efficient Structured Reasoning Paradigm
by: Yang, Kaisen, et al.
Published: (2025)

ProxSparse: Regularized Learning of Semi-Structured Sparsity Masks for Pretrained LLMs
by: Liu, Hongyi, et al.
Published: (2025)

Is In-Context Learning Sufficient for Instruction Following in LLMs?
by: Zhao, Hao, et al.
Published: (2024)

Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning
by: Li, Chengye, et al.
Published: (2025)

Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs
by: Cha, Sungmin, et al.
Published: (2024)

Let's (not) just put things in Context: Test-Time Training for Long-Context LLMs
by: Bansal, Rachit, et al.
Published: (2025)

Scaling Long-Horizon LLM Agent via Context-Folding
by: Sun, Weiwei, et al.
Published: (2025)

Why Semantic Entropy Fails: Geometry-Aware and Calibrated Uncertainty for Policy Optimization
by: Zhang, Zheyuan, et al.
Published: (2026)

An Empirical Study on Context Length for Open-Domain Dialog Generation
by: Shen, Xinyi, et al.
Published: (2024)

LongSafety: Enhance Safety for Long-Context LLMs
by: Huang, Mianqiu, et al.
Published: (2024)