Cover Image

Better Estimation of the Kullback--Leibler Divergence Between Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Amini, Afra, Vieira, Tim, Cotterell, Ryan
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Direct Preference Optimization with an Offset
by: Amini, Afra, et al.
Published: (2024)

Syntactic Control of Language Models by Posterior Inference
by: Xefteri, Vicky, et al.
Published: (2025)

Variational Best-of-N Alignment
by: Amini, Afra, et al.
Published: (2024)

Reverse-Engineering the Reader
by: Kiegeland, Samuel, et al.
Published: (2024)

Generalized Kullback-Leibler Divergence Loss
by: Cui, Jiequan, et al.
Published: (2025)

Structured Voronoi Sampling
by: Amini, Afra, et al.
Published: (2023)

Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
by: Wu, Taiqiang, et al.
Published: (2024)

Entropy and the Kullback-Leibler Divergence for Bayesian Networks: Computational Complexity and Efficient Implementation
by: Scutari, Marco
Published: (2023)

Diversity-Aware Reverse Kullback-Leibler Divergence for Large Language Model Distillation
by: Luong, Hoang-Chau, et al.
Published: (2026)

Wasserstein Distance Rivals Kullback-Leibler Divergence for Knowledge Distillation
by: Lv, Jiaming, et al.
Published: (2024)

Transformers Can Represent $n$-gram Language Models
by: Svete, Anej, et al.
Published: (2024)

Gumbel Counterfactual Generation From Language Models
by: Ravfogel, Shauli, et al.
Published: (2024)

Ensembling Language Models with Sequential Monte Carlo
by: Chan, Robin Shing Moon, et al.
Published: (2026)

The Foundations of Tokenization: Statistical and Computational Concerns
by: Gastaldi, Juan Luis, et al.
Published: (2024)

Vocabulary Expansion of Large Language Models via Kullback-Leibler-Based Self-Distillation
by: Linder, Max Rehman
Published: (2025)

Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling
by: Lipkin, Benjamin, et al.
Published: (2025)

Do Language Models Exhibit the Same Cognitive Biases in Problem Solving as Human Learners?
by: Opedal, Andreas, et al.
Published: (2024)

Language Models can Self-Improve at State-Value Estimation for Better Search
by: Mendes, Ethan, et al.
Published: (2025)

Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo
by: Loula, João, et al.
Published: (2025)

Decoupled Kullback-Leibler Divergence Loss
by: Cui, Jiequan, et al.
Published: (2023)

Assessing Large Language Models on Climate Information
by: Bulian, Jannis, et al.
Published: (2023)

Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?
by: Tang, Zilu, et al.
Published: (2025)

Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation
by: Liu, Tianyu, et al.
Published: (2024)

Instruction-tuned Language Models are Better Knowledge Learners
by: Jiang, Zhengbao, et al.
Published: (2024)

First is Not Really Better Than Last: Evaluating Layer Choice and Aggregation Strategies in Language Model Data Influence Estimation
by: Vitel, Dmytro, et al.
Published: (2025)

Are Language Models Efficient Reasoners? A Perspective from Logic Programming
by: Opedal, Andreas, et al.
Published: (2025)

Learning to Reason Efficiently with A* Post-Training
by: Opedal, Andreas, et al.
Published: (2026)

Nearly Minimax Discrete Distribution Estimation in Kullback-Leibler Divergence with High Probability
by: van der Hoeven, Dirk, et al.
Published: (2025)

Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation
by: Vu, Tu, et al.
Published: (2024)

Revisiting Padded Transformer Expressivity: Which Architectural Choices Matter and Which Don't
by: Svete, Anej, et al.
Published: (2026)

Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselves
by: Deng, Yihe, et al.
Published: (2023)

MYTE: Morphology-Driven Byte Encoding for Better and Fairer Multilingual Language Modeling
by: Limisiewicz, Tomasz, et al.
Published: (2024)

PlaSma: Making Small Language Models Better Procedural Knowledge Models for (Counterfactual) Planning
by: Brahman, Faeze, et al.
Published: (2023)

How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergence
by: Choi, Hyeong Kyu, et al.
Published: (2025)

Divergences between Language Models and Human Brains
by: Zhou, Yuchen, et al.
Published: (2023)

Principled Gradient-based Markov Chain Monte Carlo for Text Generation
by: Du, Li, et al.
Published: (2023)

Probabilistic classification from possibilistic data: computing Kullback-Leibler projection with a possibility distribution
by: Baaj, Ismaïl, et al.
Published: (2026)

Protein Language Models Diverge from Natural Language: Comparative Analysis and Improved Inference
by: Hart, Anna, et al.
Published: (2026)

Communicating Activations Between Language Model Agents
by: Ramesh, Vignav, et al.
Published: (2025)

Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct
by: Zheng, Haoyang, et al.
Published: (2025)