NeuralMatrix: Compute the Entire Neural Networks with Linear Matrix Operations for Efficient Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916362188750848 |
|---|---|
| author | Sun, Ruiqi Ye, Siwei Zhao, Jie He, Xin Lin, Jianzhe Li, Yiran Zou, An |
| author_facet | Sun, Ruiqi Ye, Siwei Zhao, Jie He, Xin Lin, Jianzhe Li, Yiran Zou, An |
| contents | The inherent diversity of computation types within the deep neural network (DNN) models often requires a variety of specialized units in hardware processors, which limits computational efficiency, increasing both inference latency and power consumption, especially when the hardware processor needs to support and execute different neural networks. In this study, we introduce NeuralMatrix, which elastically transforms the computations of entire DNNs into linear matrix operations. This transformation allows seamless execution of various DNN models all with matrix operations and paves the way for running versatile DNN models with a single General Matrix Multiplication (GEMM) accelerator.Extensive experiments with both CNN and transformer-based models demonstrate the potential of NeuralMatrix to accurately and efficiently execute a wide range of DNN models, achieving 2.17-38.72 times computation efficiency (i.e., throughput per power) compared to CPUs, GPUs, and SoC platforms. This level of efficiency is usually only attainable with the accelerator designed for a specific neural network. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_14405 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | NeuralMatrix: Compute the Entire Neural Networks with Linear Matrix Operations for Efficient Inference Sun, Ruiqi Ye, Siwei Zhao, Jie He, Xin Lin, Jianzhe Li, Yiran Zou, An Machine Learning Artificial Intelligence Hardware Architecture The inherent diversity of computation types within the deep neural network (DNN) models often requires a variety of specialized units in hardware processors, which limits computational efficiency, increasing both inference latency and power consumption, especially when the hardware processor needs to support and execute different neural networks. In this study, we introduce NeuralMatrix, which elastically transforms the computations of entire DNNs into linear matrix operations. This transformation allows seamless execution of various DNN models all with matrix operations and paves the way for running versatile DNN models with a single General Matrix Multiplication (GEMM) accelerator.Extensive experiments with both CNN and transformer-based models demonstrate the potential of NeuralMatrix to accurately and efficiently execute a wide range of DNN models, achieving 2.17-38.72 times computation efficiency (i.e., throughput per power) compared to CPUs, GPUs, and SoC platforms. This level of efficiency is usually only attainable with the accelerator designed for a specific neural network. |
| title | NeuralMatrix: Compute the Entire Neural Networks with Linear Matrix Operations for Efficient Inference |
| topic | Machine Learning Artificial Intelligence Hardware Architecture |
| url | https://arxiv.org/abs/2305.14405 |