Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912103428784128 |
|---|---|
| author | Nag, Shashank Bacellar, Alan T. L. Susskind, Zachary Jha, Anshul Liberty, Logan Sivakumar, Aishwarya John, Eugene B. Kailas, Krishnan Lima, Priscila M. V. Yadwadkar, Neeraja J. Franca, Felipe M. G. John, Lizy K. |
| author_facet | Nag, Shashank Bacellar, Alan T. L. Susskind, Zachary Jha, Anshul Liberty, Logan Sivakumar, Aishwarya John, Eugene B. Kailas, Krishnan Lima, Priscila M. V. Yadwadkar, Neeraja J. Franca, Felipe M. G. John, Lizy K. |
| contents | Transformers are set to become ubiquitous with applications ranging from chatbots and educational assistants to visual recognition and remote sensing. However, their increasing computational and memory demands is resulting in growing energy consumption. Building models with fast and energy-efficient inference is imperative to enable a variety of transformer-based applications. Look Up Table (LUT) based Weightless Neural Networks are faster than the conventional neural networks as their inference only involves a few lookup operations. Recently, an approach for learning LUT networks directly via an Extended Finite Difference method was proposed. We build on this idea, extending it for performing the functions of the Multi Layer Perceptron (MLP) layers in transformer models and integrating them with transformers to propose Quasi Weightless Transformers (QuWeiT). This allows for a computational and energy-efficient inference solution for transformer-based models. On I-ViT-T, we achieve a comparable accuracy of 95.64% on CIFAR-10 dataset while replacing approximately 55% of all the multiplications in the entire model and achieving a 2.2x energy efficiency. We also observe similar savings on experiments with the nanoGPT framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_01818 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference Nag, Shashank Bacellar, Alan T. L. Susskind, Zachary Jha, Anshul Liberty, Logan Sivakumar, Aishwarya John, Eugene B. Kailas, Krishnan Lima, Priscila M. V. Yadwadkar, Neeraja J. Franca, Felipe M. G. John, Lizy K. Machine Learning Transformers are set to become ubiquitous with applications ranging from chatbots and educational assistants to visual recognition and remote sensing. However, their increasing computational and memory demands is resulting in growing energy consumption. Building models with fast and energy-efficient inference is imperative to enable a variety of transformer-based applications. Look Up Table (LUT) based Weightless Neural Networks are faster than the conventional neural networks as their inference only involves a few lookup operations. Recently, an approach for learning LUT networks directly via an Extended Finite Difference method was proposed. We build on this idea, extending it for performing the functions of the Multi Layer Perceptron (MLP) layers in transformer models and integrating them with transformers to propose Quasi Weightless Transformers (QuWeiT). This allows for a computational and energy-efficient inference solution for transformer-based models. On I-ViT-T, we achieve a comparable accuracy of 95.64% on CIFAR-10 dataset while replacing approximately 55% of all the multiplications in the entire model and achieving a 2.2x energy efficiency. We also observe similar savings on experiments with the nanoGPT framework. |
| title | Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.01818 |