Conditional computation in neural networks: principles and research trends
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914860391989248 |
|---|---|
| author | Scardapane, Simone Baiocchi, Alessandro Devoto, Alessio Marsocci, Valerio Minervini, Pasquale Pomponi, Jary |
| author_facet | Scardapane, Simone Baiocchi, Alessandro Devoto, Alessio Marsocci, Valerio Minervini, Pasquale Pomponi, Jary |
| contents | This article summarizes principles and ideas from the emerging area of applying \textit{conditional computation} methods to the design of neural networks. In particular, we focus on neural networks that can dynamically activate or de-activate parts of their computational graph conditionally on their input. Examples include the dynamic selection of, e.g., input tokens, layers (or sets of layers), and sub-modules inside each layer (e.g., channels in a convolutional filter). We first provide a general formalism to describe these techniques in an uniform way. Then, we introduce three notable implementations of these principles: mixture-of-experts (MoEs) networks, token selection mechanisms, and early-exit neural networks. The paper aims to provide a tutorial-like introduction to this growing field. To this end, we analyze the benefits of these modular designs in terms of efficiency, explainability, and transfer learning, with a focus on emerging applicative areas ranging from automated scientific discovery to semantic communication. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07965 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Conditional computation in neural networks: principles and research trends Scardapane, Simone Baiocchi, Alessandro Devoto, Alessio Marsocci, Valerio Minervini, Pasquale Pomponi, Jary Machine Learning Artificial Intelligence This article summarizes principles and ideas from the emerging area of applying \textit{conditional computation} methods to the design of neural networks. In particular, we focus on neural networks that can dynamically activate or de-activate parts of their computational graph conditionally on their input. Examples include the dynamic selection of, e.g., input tokens, layers (or sets of layers), and sub-modules inside each layer (e.g., channels in a convolutional filter). We first provide a general formalism to describe these techniques in an uniform way. Then, we introduce three notable implementations of these principles: mixture-of-experts (MoEs) networks, token selection mechanisms, and early-exit neural networks. The paper aims to provide a tutorial-like introduction to this growing field. To this end, we analyze the benefits of these modular designs in terms of efficiency, explainability, and transfer learning, with a focus on emerging applicative areas ranging from automated scientific discovery to semantic communication. |
| title | Conditional computation in neural networks: principles and research trends |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2403.07965 |