Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.16836 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914813214457856 |
|---|---|
| author | Charalampopoulos, Andreas Chatzis, Nikolas Ntoulas-Panagiotopoulos, Foivos Papaioannou, Charilaos Potamianos, Alexandros |
| author_facet | Charalampopoulos, Andreas Chatzis, Nikolas Ntoulas-Panagiotopoulos, Foivos Papaioannou, Charilaos Potamianos, Alexandros |
| contents | Fast feedforward networks (FFFs) are a class of neural networks that exploit the observation that different regions of the input space activate distinct subsets of neurons in wide networks. FFFs partition the input space into separate sections using a differentiable binary tree of neurons and during inference descend the binary tree in order to improve computational efficiency. Inspired by Mixture of Experts (MoE) research, we propose the incorporation of load balancing and Master Leaf techniques into the FFF architecture to improve performance and simplify the training process. We reproduce experiments found in literature and present results on FFF models enhanced using these techniques. The proposed architecture and training recipe achieves up to 16.3% and 3% absolute classification accuracy increase in training and test accuracy, respectively, compared to the original FFF architecture. Additionally, we observe a smaller variance in the results compared to those reported in prior research. These findings demonstrate the potential of integrating MoE-inspired techniques into FFFs for developing more accurate and efficient models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_16836 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Fast Feed Forward Networks with Load Balancing and a Master Leaf Node Charalampopoulos, Andreas Chatzis, Nikolas Ntoulas-Panagiotopoulos, Foivos Papaioannou, Charilaos Potamianos, Alexandros Machine Learning Artificial Intelligence Fast feedforward networks (FFFs) are a class of neural networks that exploit the observation that different regions of the input space activate distinct subsets of neurons in wide networks. FFFs partition the input space into separate sections using a differentiable binary tree of neurons and during inference descend the binary tree in order to improve computational efficiency. Inspired by Mixture of Experts (MoE) research, we propose the incorporation of load balancing and Master Leaf techniques into the FFF architecture to improve performance and simplify the training process. We reproduce experiments found in literature and present results on FFF models enhanced using these techniques. The proposed architecture and training recipe achieves up to 16.3% and 3% absolute classification accuracy increase in training and test accuracy, respectively, compared to the original FFF architecture. Additionally, we observe a smaller variance in the results compared to those reported in prior research. These findings demonstrate the potential of integrating MoE-inspired techniques into FFFs for developing more accurate and efficient models. |
| title | Enhancing Fast Feed Forward Networks with Load Balancing and a Master Leaf Node |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2405.16836 |