Copertina

Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Bell, Tyler, Mudireddy, Avinash, Johnson-Eversoll, Ivan, Dasgupta, Soura, Mudumbai, Raghu
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Artificial Intelligence Information Theory
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Quantifying the perceptual value of lexical and non-lexical channels in speech
di: Wallbridge, Sarenne, et al.
Pubblicazione: (2023)

Repeated Sequences Reveal Gaps between Large Language Models and Natural Language
di: Tanaka-Ishii, Kumiko
Pubblicazione: (2026)

Large Language Models as Evaluators for Scientific Synthesis
di: Evans, Julia, et al.
Pubblicazione: (2024)

Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents
di: Wang, Haoyu, et al.
Pubblicazione: (2025)

Conversational Complexity for Assessing Risk in Large Language Models
di: Burden, John, et al.
Pubblicazione: (2024)

Confidence, Not Perplexity: A Better Metric for the Creative Era of LLMs
di: Parupudi, V. S. Raghu
Pubblicazione: (2025)

The Information of Large Language Model Geometry
di: Tan, Zhiquan, et al.
Pubblicazione: (2024)

Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graph
di: Nechakhin, Vladyslav, et al.
Pubblicazione: (2024)

TexShape: Information Theoretic Sentence Embedding for Language Models
di: Kale, Kaan, et al.
Pubblicazione: (2024)

LawLLM: Law Large Language Model for the US Legal System
di: Shu, Dong, et al.
Pubblicazione: (2024)

A Survey on Large Language Models from Concept to Implementation
di: Wang, Chen, et al.
Pubblicazione: (2024)

Feature compression is the root cause of adversarial fragility in neural network classifiers
di: Gao, Jingchao, et al.
Pubblicazione: (2024)

Large Language Models for Scientific Information Extraction: An Empirical Study for Virology
di: Shamsabadi, Mahsa, et al.
Pubblicazione: (2024)

Speculative Decoding Scaling Laws (SDSL): Throughput Optimization Made Simple
di: Bozorgkhoo, Amirhossein, et al.
Pubblicazione: (2026)

From Bayesian Asymptotics to General Large-Scale MIMO Capacity
di: Yang, Sheng, et al.
Pubblicazione: (2025)

Generalized Measures of Anticipation and Responsivity in Online Language Processing
di: Giulianelli, Mario, et al.
Pubblicazione: (2024)

Asymptotics of Language Model Alignment
di: Yang, Joy Qiping, et al.
Pubblicazione: (2024)

Task-Centric Acceleration of Small-Language Models
di: Tsur, Dor, et al.
Pubblicazione: (2026)

Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models
di: Wei, Lai, et al.
Pubblicazione: (2024)

Think Less, Label Better: Multi-Stage Domain-Grounded Synthetic Data Generation for Fine-Tuning Large Language Models in Telecommunications
di: Shi, Chenhua, et al.
Pubblicazione: (2025)

An Enhanced Text Compression Approach Using Transformer-based Language Models
di: Rahman, Chowdhury Mofizur, et al.
Pubblicazione: (2024)

Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lens
di: Yong, Xixian, et al.
Pubblicazione: (2025)

Optimal Multi-bit Generative Watermarking Schemes Under Worst-Case False-Alarm Constraints
di: Huang, Yu-Shin, et al.
Pubblicazione: (2026)

A Law of Large Numbers with Convergence Rate based on Nonlinear Expectation Theory and Its Application to Communication Detection
di: Fu, Jialiang, et al.
Pubblicazione: (2026)

L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling
di: Chen, Zhuo, et al.
Pubblicazione: (2025)

Fundamental Limits of Prompt Compression: A Rate-Distortion Framework for Black-Box Language Models
di: Nagle, Alliot, et al.
Pubblicazione: (2024)

Language Modeling Is Compression
di: Delétang, Grégoire, et al.
Pubblicazione: (2023)

Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models
di: Klein, Tassilo, et al.
Pubblicazione: (2024)

Asymptotically Optimal Sequence Sets With Low/Zero Ambiguity Zone Properties
di: Tian, Liying, et al.
Pubblicazione: (2024)

Information-Theoretic Generative Clustering of Documents
di: Du, Xin, et al.
Pubblicazione: (2024)

Nacrith: Neural Lossless Compression via Ensemble Context Modeling and High-Precision CDF Coding
di: Tacconelli, Roberto
Pubblicazione: (2026)

Theoretical Limits of Language Model Alignment
di: Paes, Lucas Monteiro, et al.
Pubblicazione: (2026)

Precise Asymptotics for Spectral Methods in Mixed Generalized Linear Models
di: Zhang, Yihan, et al.
Pubblicazione: (2022)

Cross-Entropy Games for Language Models: From Implicit Knowledge to General Capability Measures
di: Hongler, Clément, et al.
Pubblicazione: (2025)

Hessian of Perplexity for Large Language Models by PyTorch autograd (Open Source)
di: Ilin, Ivan
Pubblicazione: (2025)

Generative Multi-Modal Knowledge Retrieval with Large Language Models
di: Long, Xinwei, et al.
Pubblicazione: (2024)

Efficient Inference for Large Language Model-based Generative Recommendation
di: Lin, Xinyu, et al.
Pubblicazione: (2024)

Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models
di: Li, Gen, et al.
Pubblicazione: (2023)

Measuring Grammatical Diversity from Small Corpora: Derivational Entropy Rates, Mean Length of Utterances, and Annotation Invariance
di: Martin, Fermin Moscoso del Prado
Pubblicazione: (2024)

A Mathematical Theory for Learning Semantic Languages by Abstract Learners
di: Liao, Kuo-Yu, et al.
Pubblicazione: (2024)