1-Lipschitz Network Initialization for Certifiably Robust Classification Applications: A Decay Problem
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915624858419200 |
|---|---|
| author | Juston, Marius F. R. Sreenivas, Ramavarapu S. Norris, William R. Nottage, Dustin Soylemezoglu, Ahmet |
| author_facet | Juston, Marius F. R. Sreenivas, Ramavarapu S. Norris, William R. Nottage, Dustin Soylemezoglu, Ahmet |
| contents | This paper discusses the weight parametrization of two standard 1-Lipschitz network architectures, the Almost-Orthogonal-Layers (AOL) and the SDP-based Lipschitz Layers (SLL). It examines their impact on initialization for deep 1-Lipschitz feedforward networks, and discusses underlying issues surrounding this initialization. These networks are mainly used in certifiably robust classification applications to combat adversarial attacks by limiting the impact of perturbations on the classification output. Exact and upper bounds for the parameterized weight variance were calculated assuming a standard Normal distribution initialization; additionally, an upper bound was computed assuming a Generalized Normal Distribution, generalizing the proof for Uniform, Laplace, and Normal distribution weight initializations. It is demonstrated that the weight variance holds no bearing on the output variance distribution and that only the dimension of the weight matrices matters. Additionally, this paper demonstrates that the weight initialization always causes deep 1-Lipschitz networks to decay to zero. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_00240 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | 1-Lipschitz Network Initialization for Certifiably Robust Classification Applications: A Decay Problem Juston, Marius F. R. Sreenivas, Ramavarapu S. Norris, William R. Nottage, Dustin Soylemezoglu, Ahmet Machine Learning Artificial Intelligence This paper discusses the weight parametrization of two standard 1-Lipschitz network architectures, the Almost-Orthogonal-Layers (AOL) and the SDP-based Lipschitz Layers (SLL). It examines their impact on initialization for deep 1-Lipschitz feedforward networks, and discusses underlying issues surrounding this initialization. These networks are mainly used in certifiably robust classification applications to combat adversarial attacks by limiting the impact of perturbations on the classification output. Exact and upper bounds for the parameterized weight variance were calculated assuming a standard Normal distribution initialization; additionally, an upper bound was computed assuming a Generalized Normal Distribution, generalizing the proof for Uniform, Laplace, and Normal distribution weight initializations. It is demonstrated that the weight variance holds no bearing on the output variance distribution and that only the dimension of the weight matrices matters. Additionally, this paper demonstrates that the weight initialization always causes deep 1-Lipschitz networks to decay to zero. |
| title | 1-Lipschitz Network Initialization for Certifiably Robust Classification Applications: A Decay Problem |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2503.00240 |