Cover Image

L$^2$M: Mutual Information Scaling Law for Long-Context Language Modeling

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Zhuo, Comas, Oriol Mayné i, Jin, Zhuotao, Luo, Di, Soljačić, Marin
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence Information Theory Machine Learning Data Analysis, Statistics and Probability
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Accurate Estimation of Mutual Information in High Dimensional Data
by: Abdelaleem, Eslam, et al.
Published: (2025)

Exact Identity Linking Entropy Production and Mutual Information
by: Cho, Doohyeong, et al.
Published: (2025)

Dynamics of Collective Information Processing for Risk Encoding in Social Networks during Crises
by: Fan, Chao, et al.
Published: (2024)

Universal Scaling Laws in Freeway Traffic
by: Lee, Garyoung, et al.
Published: (2025)

Noise-Robust One-Bit Diffraction Tomography and Optimal Dose Fractionation
by: Chen, Pengwen, et al.
Published: (2023)

Trajectory analysis through entropy characterization over coded representation
by: Peña-Mendieta, Roxana, et al.
Published: (2024)

Physics-Aware, Shannon-Optimal Compression via Arithmetic Coding for Distributional Fidelity
by: Fanelli, Cristiano
Published: (2026)

Characterising high-order interdependence via entropic conjugation
by: Rosas, Fernando E., et al.
Published: (2024)

A totally empirical basis of science
by: Loukas, Orestis, et al.
Published: (2024)

Message-Passing on Hypergraphs: Detectability, Phase Transitions and Higher-Order Information
by: Ruggeri, Nicolò, et al.
Published: (2023)

A Thermodynamic Structure of Asymptotic Inference
by: Wong, Willy
Published: (2026)

First-Extinction Law for Resampling Processes
by: Benati, Matteo, et al.
Published: (2025)

Information Theory for Complex Systems Scientists
by: Varley, Thomas F.
Published: (2023)

Deep Variational Multivariate Information Bottleneck -- A Framework for Variational Losses
by: Abdelaleem, Eslam, et al.
Published: (2023)

TPCpp-10M: Simulated proton-proton collisions in a Time Projection Chamber for AI Foundation Models
by: Li, Shuhang, et al.
Published: (2025)

Information-theoretic analysis of temporal dependence in discrete stochastic processes: Application to precipitation predictability
by: De Gregorio, Juan, et al.
Published: (2025)

Information bottleneck for learning the phase space of dynamics from high-dimensional experimental data
by: Martini, K. Michael, et al.
Published: (2026)

Modular Debiasing: A Robust Method for Quantum Randomness Extraction
by: Gueron, Eduardo
Published: (2025)

Mutual information and task-relevant latent dimensionality
by: Gulati, Paarth, et al.
Published: (2026)

Iterative structural coarse-graining for contagion dynamics in complex networks
by: Xue, Leyang, et al.
Published: (2024)

An egonet-based approach to effective weighted network comparison
by: Piccardi, Carlo
Published: (2024)

On the reconstruction limits of complex networks
by: Murphy, Charles, et al.
Published: (2024)

Overlapping community detection in weighted networks
by: Qing, Huan
Published: (2022)

mdendro: An R package for extended agglomerative hierarchical clustering
by: Fernández, Alberto, et al.
Published: (2023)

Parsimonious Hawkes Processes for temporal networks modelling
by: Zhu, Yuwei, et al.
Published: (2025)

Comparing discriminating abilities of evaluation metrics in link prediction
by: Jiao, Xinshan, et al.
Published: (2024)

Diffusion Signals Reveal Hidden Connections: A Physics-Inspired Framework for Link Prediction via Personalized PageRank Signals
by: Deng, Huilin Wang Wenjun Zhang Weibing
Published: (2025)

Random walks with resetting on hypergraph
by: Ma, Fei, et al.
Published: (2025)

Higher-order null models as a lens for social systems
by: Preti, Giulia, et al.
Published: (2024)

Estimating Nodal Spreading Influence Using Partial Temporal Network
by: Mao, Tianrui, et al.
Published: (2025)

Transfer entropy for finite data
by: Kirkley, Alec
Published: (2025)

Precise influence evaluation in complex networks
by: Zhu, Bingyu, et al.
Published: (2023)

Identifying preferred routes of sharing information on social networks
by: Mohammadikian, Rozhin, et al.
Published: (2025)

A feature-based information-theoretic approach for detecting interpretable, long-timescale pairwise interactions from time series
by: Nguyen, Aria, et al.
Published: (2024)

Scalar embedding of temporal network trajectories
by: Lacasa, Lucas, et al.
Published: (2024)

Entropy-based models to randomize real-world hypergraphs
by: Saracco, Fabio, et al.
Published: (2022)

Generating Networks to Target Assortativity via Archimedean Copula Graphons
by: Idowu, Victory
Published: (2025)

Network Sampling: An Overview and Comparative Analysis
by: Nguyen, Quoc Chuong
Published: (2025)

Mutual information as a measure of mixing efficiency in viscous fluids
by: Shi, Yihong, et al.
Published: (2023)

Human mobility is well described by closed-form gravity-like models learned automatically from data
by: Cabanas-Tirapu, Oriol, et al.
Published: (2023)