Cover Image

How Many Features Can a Language Model Store Under the Linear Representation Hypothesis?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Garg, Nikhil, Kleinberg, Jon, Peng, Kenny
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence Computation and Language Information Theory Combinatorics
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

A No Free Lunch Theorem for Human-AI Collaboration
by: Peng, Kenny, et al.
Published: (2024)

Use Sparse Autoencoders to Discover Unknown Concepts, Not to Act on Known Concepts
by: Peng, Kenny, et al.
Published: (2025)

Sparse Autoencoders for Hypothesis Generation
by: Movva, Rajiv, et al.
Published: (2025)

Correlated Errors in Large Language Models
by: Kim, Elliot, et al.
Published: (2025)

Density Measures for Language Generation
by: Kleinberg, Jon, et al.
Published: (2025)

The Linear Representation Hypothesis and the Geometry of Large Language Models
by: Park, Kiho, et al.
Published: (2023)

The Detection-Extraction Gap: Models Know the Answer Before They Can Say It
by: Wang, Hanyang, et al.
Published: (2026)

Tele-LLMs: A Series of Specialized Large Language Models for Telecommunications
by: Maatouk, Ali, et al.
Published: (2024)

Compression Represents Intelligence Linearly
by: Huang, Yuzhen, et al.
Published: (2024)

Language Modeling Is Compression
by: Delétang, Grégoire, et al.
Published: (2023)

The Information of Large Language Model Geometry
by: Tan, Zhiquan, et al.
Published: (2024)

Language Generation in the Limit
by: Kleinberg, Jon, et al.
Published: (2024)

SPEX: Scaling Feature Interaction Explanations for LLMs
by: Kang, Justin Singh, et al.
Published: (2025)

A Survey on Large Language Models from Concept to Implementation
by: Wang, Chen, et al.
Published: (2024)

Geometric Signatures of Compositionality Across a Language Model's Lifetime
by: Lee, Jin Hwa, et al.
Published: (2024)

Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models
by: Wei, Lai, et al.
Published: (2024)

Disentangled Structural and Featural Representation for Task-Agnostic Graph Valuation
by: Falahati, Ali, et al.
Published: (2024)

Can Kernel Methods Explain How the Data Affects Neural Collapse?
by: Kothapalli, Vignesh, et al.
Published: (2024)

Linear Representation Transferability Hypothesis: Leveraging Small Models to Steer Large Models
by: Bello, Femi, et al.
Published: (2025)

Microstructures and Accuracy of Graph Recall by Large Language Models
by: Wang, Yanbang, et al.
Published: (2024)

Know Your Limits: Entropy Estimation Modeling for Compression and Generalization
by: Badger, Benjamin L., et al.
Published: (2025)

Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face
by: Laufer, Benjamin, et al.
Published: (2025)

Scaling Laws for Many-Shot In-Context Learning with Self-Generated Annotations
by: Gu, Zhengyao, et al.
Published: (2025)

Delta Activations: A Representation for Finetuned Large Language Models
by: Xu, Zhiqiu, et al.
Published: (2025)

How Random is Random? Evaluating the Randomness and Humaness of LLMs' Coin Flips
by: Van Koevering, Katherine, et al.
Published: (2024)

On Language Generation in the Limit with Bounded Memory
by: Kleinberg, Jon, et al.
Published: (2026)

NanoKnow: How to Know What Your Language Model Knows
by: Gu, Lingwei, et al.
Published: (2026)

Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
by: Ye, Jiasheng, et al.
Published: (2023)

Language Models Can Reduce Asymmetry in Information Markets
by: Rahaman, Nasim, et al.
Published: (2024)

A Training-free Method for LLM Text Attribution
by: Radvand, Tara, et al.
Published: (2025)

Self-Play Only Evolves When Self-Synthetic Pipeline Ensures Learnable Information Gain
by: Liu, Wei, et al.
Published: (2026)

Analyzing and Improving Chain-of-Thought Monitorability Through Information Theory
by: Anwar, Usman, et al.
Published: (2026)

Optimal Quantization for Matrix Multiplication
by: Ordentlich, Or, et al.
Published: (2024)

Memorization-Compression Cycles Improve Generalization
by: Yu, Fangyuan
Published: (2025)

Measuring Uncertainty in Transformer Circuits with Effective Information Consistency
by: Krasnovsky, Anatoly A.
Published: (2025)

A Communication-Theoretic Framework for LLM Agents: Cost-Aware Adaptive Reliability
by: Omidvar, Hamed, et al.
Published: (2026)

The Stepwise Informativeness Assumption: Why are Entropy Dynamics and Reasoning Correlated in LLMs?
by: Català, Mar Gonzàlez I, et al.
Published: (2026)

Learning is Forgetting: LLM Training As Lossy Compression
by: Conklin, Henry C., et al.
Published: (2026)

Subjective Depth and Timescale Transformers: Learning Where and When to Compute
by: Wieser, Frederico, et al.
Published: (2025)

SQuat: Subspace-orthogonal KV Cache Quantization
by: Wang, Hao, et al.
Published: (2025)