Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Súkeník, Peter, Mondelli, Marco, Lampert, Christoph
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Optimization and Control
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866914979569991680
author	Súkeník, Peter Mondelli, Marco Lampert, Christoph
author_facet	Súkeník, Peter Mondelli, Marco Lampert, Christoph
contents	Deep neural networks (DNNs) exhibit a surprising structure in their final layer known as neural collapse (NC), and a growing body of works has currently investigated the propagation of neural collapse to earlier layers of DNNs -- a phenomenon called deep neural collapse (DNC). However, existing theoretical results are restricted to special cases: linear models, only two layers or binary classification. In contrast, we focus on non-linear models of arbitrary depth in multi-class classification and reveal a surprising qualitative shift. As soon as we go beyond two layers or two classes, DNC stops being optimal for the deep unconstrained features model (DUFM) -- the standard theoretical framework for the analysis of collapse. The main culprit is a low-rank bias of multi-layer regularization schemes: this bias leads to optimal solutions of even lower rank than the neural collapse. We support our theoretical findings with experiments on both DUFM and real data, which show the emergence of the low-rank structure in the solution found by gradient descent.
format	Preprint
id	arxiv_https___arxiv_org_abs_2405_14468
institution	arXiv
publishDate	2024
record_format	arxiv
spellingShingle	Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal? Súkeník, Peter Mondelli, Marco Lampert, Christoph Machine Learning Optimization and Control Deep neural networks (DNNs) exhibit a surprising structure in their final layer known as neural collapse (NC), and a growing body of works has currently investigated the propagation of neural collapse to earlier layers of DNNs -- a phenomenon called deep neural collapse (DNC). However, existing theoretical results are restricted to special cases: linear models, only two layers or binary classification. In contrast, we focus on non-linear models of arbitrary depth in multi-class classification and reveal a surprising qualitative shift. As soon as we go beyond two layers or two classes, DNC stops being optimal for the deep unconstrained features model (DUFM) -- the standard theoretical framework for the analysis of collapse. The main culprit is a low-rank bias of multi-layer regularization schemes: this bias leads to optimal solutions of even lower rank than the neural collapse. We support our theoretical findings with experiments on both DUFM and real data, which show the emergence of the low-rank structure in the solution found by gradient descent.
title	Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?
topic	Machine Learning Optimization and Control
url	https://arxiv.org/abs/2405.14468