Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916804608131072 |
|---|---|
| author | Park, Sangyeon Han, Isaac Oh, Seungwon Kim, Kyung-Joong |
| author_facet | Park, Sangyeon Han, Isaac Oh, Seungwon Kim, Kyung-Joong |
| contents | Plasticity loss, a critical challenge in neural network training, limits a model's ability to adapt to new tasks or shifts in data distribution. This paper introduces AID (Activation by Interval-wise Dropout), a novel method inspired by Dropout, designed to address plasticity loss. Unlike Dropout, AID generates subnetworks by applying Dropout with different probabilities on each preactivation interval. Theoretical analysis reveals that AID regularizes the network, promoting behavior analogous to that of deep linear networks, which do not suffer from plasticity loss. We validate the effectiveness of AID in maintaining plasticity across various benchmarks, including continual learning tasks on standard image classification datasets such as CIFAR10, CIFAR100, and TinyImageNet. Furthermore, we show that AID enhances reinforcement learning performance in the Arcade Learning Environment benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_01342 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss Park, Sangyeon Han, Isaac Oh, Seungwon Kim, Kyung-Joong Machine Learning Artificial Intelligence Plasticity loss, a critical challenge in neural network training, limits a model's ability to adapt to new tasks or shifts in data distribution. This paper introduces AID (Activation by Interval-wise Dropout), a novel method inspired by Dropout, designed to address plasticity loss. Unlike Dropout, AID generates subnetworks by applying Dropout with different probabilities on each preactivation interval. Theoretical analysis reveals that AID regularizes the network, promoting behavior analogous to that of deep linear networks, which do not suffer from plasticity loss. We validate the effectiveness of AID in maintaining plasticity across various benchmarks, including continual learning tasks on standard image classification datasets such as CIFAR10, CIFAR100, and TinyImageNet. Furthermore, we show that AID enhances reinforcement learning performance in the Arcade Learning Environment benchmark. |
| title | Activation by Interval-wise Dropout: A Simple Way to Prevent Neural Networks from Plasticity Loss |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.01342 |