Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.01263 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912622256848896 |
|---|---|
| author | Meirovitch, Yaron Yang, Fuming Lichtman, Jeff Shavit, Nir |
| author_facet | Meirovitch, Yaron Yang, Fuming Lichtman, Jeff Shavit, Nir |
| contents | Most pruning methods remove parameters ranked by impact on loss (e.g., magnitude or gradient). We propose Budgeted Broadcast (BB), which gives each unit a local traffic budget (the product of its long-term on-rate $a_i$ and fan-out $k_i$). A constrained-entropy analysis shows that maximizing coding entropy under a global traffic budget yields a selectivity-audience balance, $\log\frac{1-a_i}{a_i}=βk_i$. BB enforces this balance with simple local actuators that prune either fan-in (to lower activity) or fan-out (to reduce broadcast). In practice, BB increases coding entropy and decorrelation and improves accuracy at matched sparsity across Transformers for ASR, ResNets for face identification, and 3D U-Nets for synapse prediction, sometimes exceeding dense baselines. On electron microscopy images, it attains state-of-the-art F1 and PR-AUC under our evaluation protocol. BB is easy to integrate and suggests a path toward learning more diverse and efficient representations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_01263 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Budgeted Broadcast: An Activity-Dependent Pruning Rule for Neural Network Efficiency Meirovitch, Yaron Yang, Fuming Lichtman, Jeff Shavit, Nir Machine Learning Artificial Intelligence Most pruning methods remove parameters ranked by impact on loss (e.g., magnitude or gradient). We propose Budgeted Broadcast (BB), which gives each unit a local traffic budget (the product of its long-term on-rate $a_i$ and fan-out $k_i$). A constrained-entropy analysis shows that maximizing coding entropy under a global traffic budget yields a selectivity-audience balance, $\log\frac{1-a_i}{a_i}=βk_i$. BB enforces this balance with simple local actuators that prune either fan-in (to lower activity) or fan-out (to reduce broadcast). In practice, BB increases coding entropy and decorrelation and improves accuracy at matched sparsity across Transformers for ASR, ResNets for face identification, and 3D U-Nets for synapse prediction, sometimes exceeding dense baselines. On electron microscopy images, it attains state-of-the-art F1 and PR-AUC under our evaluation protocol. BB is easy to integrate and suggests a path toward learning more diverse and efficient representations. |
| title | Budgeted Broadcast: An Activity-Dependent Pruning Rule for Neural Network Efficiency |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.01263 |