From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916827242692608 |
|---|---|
| author | Bouniot, Quentin Redko, Ievgen Mallasto, Anton Laclau, Charlotte Struckmeier, Oliver Arndt, Karol Heinonen, Markus Kyrki, Ville Kaski, Samuel |
| author_facet | Bouniot, Quentin Redko, Ievgen Mallasto, Anton Laclau, Charlotte Struckmeier, Oliver Arndt, Karol Heinonen, Markus Kyrki, Ville Kaski, Samuel |
| contents | In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, with a particular emphasis on the computer vision task. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature and its potential for long-reaching implications. The code for our work is available at https://github.com/qbouniot/AffScoreDeep |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_11439 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport Bouniot, Quentin Redko, Ievgen Mallasto, Anton Laclau, Charlotte Struckmeier, Oliver Arndt, Karol Heinonen, Markus Kyrki, Ville Kaski, Samuel Machine Learning Artificial Intelligence In the last decade, we have witnessed the introduction of several novel deep neural network (DNN) architectures exhibiting ever-increasing performance across diverse tasks. Explaining the upward trend of their performance, however, remains difficult as different DNN architectures of comparable depth and width -- common factors associated with their expressive power -- may exhibit a drastically different performance even when trained on the same dataset. In this paper, we introduce the concept of the non-linearity signature of DNN, the first theoretically sound solution for approximately measuring the non-linearity of deep neural networks. Built upon a score derived from closed-form optimal transport mappings, this signature provides a better understanding of the inner workings of a wide range of DNN architectures and learning paradigms, with a particular emphasis on the computer vision task. We provide extensive experimental results that highlight the practical usefulness of the proposed non-linearity signature and its potential for long-reaching implications. The code for our work is available at https://github.com/qbouniot/AffScoreDeep |
| title | From Alexnet to Transformers: Measuring the Non-linearity of Deep Neural Networks with Affine Optimal Transport |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2310.11439 |