MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916449596997632 |
|---|---|
| author | Yang, Zebin Chen, Renze Wu, Taiqiang Wong, Ngai Liang, Yun Wang, Runsheng Huang, Ru Li, Meng |
| author_facet | Yang, Zebin Chen, Renze Wu, Taiqiang Wong, Ngai Liang, Yun Wang, Runsheng Huang, Ru Li, Meng |
| contents | In this paper, we propose MCUBERT to enable language models like BERT on tiny microcontroller units (MCUs) through network and scheduling co-optimization. We observe the embedding table contributes to the major storage bottleneck for tiny BERT models. Hence, at the network level, we propose an MCU-aware two-stage neural architecture search algorithm based on clustered low-rank approximation for embedding compression. To reduce the inference memory requirements, we further propose a novel fine-grained MCU-friendly scheduling strategy. Through careful computation tiling and re-ordering as well as kernel design, we drastically increase the input sequence lengths supported on MCUs without any latency or accuracy penalty. MCUBERT reduces the parameter size of BERT-tiny and BERT-mini by 5.7$\times$ and 3.0$\times$ and the execution memory by 3.5$\times$ and 4.3$\times$, respectively. MCUBERT also achieves 1.5$\times$ latency reduction. For the first time, MCUBERT enables lightweight BERT models on commodity MCUs and processing more than 512 tokens with less than 256KB of memory. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_17957 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers Yang, Zebin Chen, Renze Wu, Taiqiang Wong, Ngai Liang, Yun Wang, Runsheng Huang, Ru Li, Meng Machine Learning Artificial Intelligence In this paper, we propose MCUBERT to enable language models like BERT on tiny microcontroller units (MCUs) through network and scheduling co-optimization. We observe the embedding table contributes to the major storage bottleneck for tiny BERT models. Hence, at the network level, we propose an MCU-aware two-stage neural architecture search algorithm based on clustered low-rank approximation for embedding compression. To reduce the inference memory requirements, we further propose a novel fine-grained MCU-friendly scheduling strategy. Through careful computation tiling and re-ordering as well as kernel design, we drastically increase the input sequence lengths supported on MCUs without any latency or accuracy penalty. MCUBERT reduces the parameter size of BERT-tiny and BERT-mini by 5.7$\times$ and 3.0$\times$ and the execution memory by 3.5$\times$ and 4.3$\times$, respectively. MCUBERT also achieves 1.5$\times$ latency reduction. For the first time, MCUBERT enables lightweight BERT models on commodity MCUs and processing more than 512 tokens with less than 256KB of memory. |
| title | MCUBERT: Memory-Efficient BERT Inference on Commodity Microcontrollers |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.17957 |