A Theoretical View of Linear Backpropagation and Its Convergence
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2021
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911752648654848 |
|---|---|
| author | Li, Ziang Guo, Yiwen Liu, Haodi Zhang, Changshui |
| author_facet | Li, Ziang Guo, Yiwen Liu, Haodi Zhang, Changshui |
| contents | Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs). Applied often along with stochastic gradient descent (SGD) or its variants, BP is considered as a de-facto choice in a variety of machine learning tasks including DNN training and adversarial attack/defense. Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks, by Guo et al. Although it has been shown empirically effective in black-box attacks, theoretical studies and convergence analyses of such a method is lacking. This paper serves as a complement and somewhat an extension to Guo et al.'s paper, by providing theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training. We demonstrate that, somewhat surprisingly, LinBP can lead to faster convergence in these tasks in the same hyper-parameter settings, compared to BP. We confirm our theoretical results with extensive experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2112_11018 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | A Theoretical View of Linear Backpropagation and Its Convergence Li, Ziang Guo, Yiwen Liu, Haodi Zhang, Changshui Machine Learning Cryptography and Security Computer Vision and Pattern Recognition Neural and Evolutionary Computing Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs). Applied often along with stochastic gradient descent (SGD) or its variants, BP is considered as a de-facto choice in a variety of machine learning tasks including DNN training and adversarial attack/defense. Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks, by Guo et al. Although it has been shown empirically effective in black-box attacks, theoretical studies and convergence analyses of such a method is lacking. This paper serves as a complement and somewhat an extension to Guo et al.'s paper, by providing theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training. We demonstrate that, somewhat surprisingly, LinBP can lead to faster convergence in these tasks in the same hyper-parameter settings, compared to BP. We confirm our theoretical results with extensive experiments. |
| title | A Theoretical View of Linear Backpropagation and Its Convergence |
| topic | Machine Learning Cryptography and Security Computer Vision and Pattern Recognition Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2112.11018 |