Imagen de Portada

An Information-Theoretic Approach to Understanding Transformers' In-Context Learning of Variable-Order Markov Chains

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Zhou, Ruida, Tian, Chao, Diggavi, Suhas
Formato:	Preprint
Publicado:	2024
Materias:	Machine Learning Information Theory
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Robust Federated Personalised Mean Estimation for the Gaussian Mixture Model
por: Managoli, Malhar A., et al.
Publicado: (2025)

ADEPT: Hierarchical Bayes Approach to Personalized Federated Unsupervised Learning
por: Ozkara, Kaan, et al.
Publicado: (2024)

On the Training Convergence of Transformers for In-Context Classification of Gaussian Mixtures
por: Shen, Wei, et al.
Publicado: (2024)

SPIRE: Conditional Personalization for Federated Diffusion Generative Models
por: Ozkara, Kaan, et al.
Publicado: (2025)

From Markov to Laplace: How Mamba In-Context Learns Markov Chains
por: Bondaschi, Marco, et al.
Publicado: (2025)

An Information-Theoretic Analysis of In-Context Learning
por: Jeon, Hong Jun, et al.
Publicado: (2024)

Reframing Data Value for Large Language Models Through the Lens of Plausibility
por: Rammal, Mohamad Rida, et al.
Publicado: (2024)

Information-Theoretic Framework for Understanding Modern Machine-Learning
por: Feder, Meir, et al.
Publicado: (2025)

On the optimal regret of collaborative personalized linear bandits
por: Huang, Bruce, et al.
Publicado: (2025)

Context Channel Capacity: An Information-Theoretic Framework for Understanding Catastrophic Forgetting
por: Cheng, Ran
Publicado: (2026)

Information-Theoretic Measures on Lattices for High-Order Interactions
por: Liu, Zhaolu, et al.
Publicado: (2024)

Information-Theoretic State Variable Selection for Reinforcement Learning
por: Westphal, Charles, et al.
Publicado: (2024)

Attention with Markov: A Framework for Principled Analysis of Transformers via Markov Chains
por: Makkuva, Ashok Vardhan, et al.
Publicado: (2024)

Directed Information $γ$-covering: An Information-Theoretic Framework for Context Engineering
por: Huang, Hai
Publicado: (2025)

Common Information Dimension
por: Hanna, Osama, et al.
Publicado: (2023)

Cost-Aware Optimal Pairwise Pure Exploration
por: Wu, Di, et al.
Publicado: (2025)

Harnessing the Power of Federated Learning in Federated Contextual Bandits
por: Shi, Chengshuai, et al.
Publicado: (2023)

Information Theoretic Perspective on Representation Learning
por: Pereg, Deborah, et al.
Publicado: (2026)

Provable Policy Gradient Methods for Average-Reward Markov Potential Games
por: Cheng, Min, et al.
Publicado: (2024)

On the Generalization for Transfer Learning: An Information-Theoretic Analysis
por: Wu, Xuetong, et al.
Publicado: (2022)

Near-Optimal Clustering in Mixture of Markov Chains
por: Lee, Junghyun, et al.
Publicado: (2025)

Staggered Quantizers for Perfect Perceptual Quality: A Connection between Quantizers with Common Randomness and Without
por: Zhou, Ruida, et al.
Publicado: (2024)

Information Theoretic Learning for Diffusion Models with Warm Start
por: Shen, Yirong, et al.
Publicado: (2025)

Improved Information Theoretic Generalization Bounds for Distributed and Federated Learning
por: Barnes, L. P., et al.
Publicado: (2022)

Information-Theoretic Minimax Regret Bounds for Reinforcement Learning based on Duality
por: Bongole, Raghav, et al.
Publicado: (2024)

Information-Theoretic Discrete Diffusion
por: Jeon, Moongyu, et al.
Publicado: (2025)

An Information Theoretic Perspective on Conformal Prediction
por: Correia, Alvaro H. C., et al.
Publicado: (2024)

An Information-Theoretic Analysis of Temporal GNNs
por: Farzaneh, Amirmohammad
Publicado: (2024)

Information-Theoretic Proofs for Diffusion Sampling
por: Reeves, Galen, et al.
Publicado: (2025)

On Causality in Domain Adaptation and Semi-Supervised Learning: an Information-Theoretic Analysis for Parametric Models
por: Wu, Xuetong, et al.
Publicado: (2022)

On the Generalization of Knowledge Distillation: An Information-Theoretic View
por: Li, Bingying, et al.
Publicado: (2026)

Understanding Encoder-Decoder Structures in Machine Learning Using Information Measures
por: Silva, Jorge F., et al.
Publicado: (2024)

Information-Theoretic Generalization Bounds for Deep Neural Networks
por: He, Haiyun, et al.
Publicado: (2024)

Unifying Information-Theoretic and Pair-Counting Clustering Similarity
por: Gates, Alexander J.
Publicado: (2025)

Transformers on Markov Data: Constant Depth Suffices
por: Rajaraman, Nived, et al.
Publicado: (2024)

Broadcast Channel Cooperative Gain: An Operational Interpretation of Partial Information Decomposition
por: Tian, Chao, et al.
Publicado: (2025)

Fairness Overfitting in Machine Learning: An Information-Theoretic Perspective
por: Laakom, Firas, et al.
Publicado: (2025)

On Theoretical Limits of Learning with Label Differential Privacy
por: Zhao, Puning, et al.
Publicado: (2025)

Information Theoretic Guarantees For Policy Alignment In Large Language Models
por: Mroueh, Youssef
Publicado: (2024)

Information-Theoretic Thresholds for the Alignments of Partially Correlated Graphs
por: Huang, Dong, et al.
Publicado: (2024)