The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913098855612416 |
|---|---|
| author | Cha, Taehun Beaglehole, Daniel Radhakrishnan, Adityanarayanan Lee, Donghun |
| author_facet | Cha, Taehun Beaglehole, Daniel Radhakrishnan, Adityanarayanan Lee, Donghun |
| contents | Understanding how deep neural networks learn representations remains a central challenge in machine learning theory. In this work, we propose a feature-centric framework for analyzing neural network training by relating weight updates to feature evolution. We introduce a simple identity, the Feature Learning Equation, which identifies the weight Gram matrix as the key object capturing feature dynamics. This enables us to interpret gradient descent as implicitly inducing a hypothetical evolution of features, whose covariance structure - termed the Virtual Covariance - characterizes how representations evolve during training. Building on this perspective, we introduce Target Linearity, a measure quantifying the linear alignment between features and targets. By analyzing the training and layer-wise dynamics, we show that deep networks learn to sequentially transform representations toward target-linear structure. This linearization perspective provides a unified interpretation of several empirical phenomena, including Neural Collapse and linear interpolation in generative models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_06258 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks Cha, Taehun Beaglehole, Daniel Radhakrishnan, Adityanarayanan Lee, Donghun Machine Learning Artificial Intelligence Understanding how deep neural networks learn representations remains a central challenge in machine learning theory. In this work, we propose a feature-centric framework for analyzing neural network training by relating weight updates to feature evolution. We introduce a simple identity, the Feature Learning Equation, which identifies the weight Gram matrix as the key object capturing feature dynamics. This enables us to interpret gradient descent as implicitly inducing a hypothetical evolution of features, whose covariance structure - termed the Virtual Covariance - characterizes how representations evolve during training. Building on this perspective, we introduce Target Linearity, a measure quantifying the linear alignment between features and targets. By analyzing the training and layer-wise dynamics, we show that deep networks learn to sequentially transform representations toward target-linear structure. This linearization perspective provides a unified interpretation of several empirical phenomena, including Neural Collapse and linear interpolation in generative models. |
| title | The Weight Gram Matrix Captures Sequential Feature Linearization in Deep Networks |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.06258 |