First Demonstration of Second-order Training of Deep Neural Networks with In-memory Analog Matrix Computing
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908694803906560 |
|---|---|
| author | Zhang, Saitao Luo, Yubiao Wang, Shiqing Zuo, Pushen Li, Yongxiang Pan, Lunshuai Miao, Zheng Sun, Zhong |
| author_facet | Zhang, Saitao Luo, Yubiao Wang, Shiqing Zuo, Pushen Li, Yongxiang Pan, Lunshuai Miao, Zheng Sun, Zhong |
| contents | Second-order optimization methods, which leverage curvature information, offer faster and more stable convergence than first-order methods such as stochastic gradient descent (SGD) and Adam. However, their practical adoption is hindered by the prohibitively high cost of inverting the second-order information matrix, particularly in large-scale neural network training. Here, we present the first demonstration of a second-order optimizer powered by in-memory analog matrix computing (AMC) using resistive random-access memory (RRAM), which performs matrix inversion (INV) in a single step. We validate the optimizer by training a two-layer convolutional neural network (CNN) for handwritten letter classification, achieving 26% and 61% fewer training epochs than SGD with momentum and Adam, respectively. On a larger task using the same second-order method, our system delivers a 5.88x improvement in throughput and a 6.9x gain in energy efficiency compared to state-of-the-art digital processors. These results demonstrate the feasibility and effectiveness of AMC circuits for second-order neural network training, opening a new path toward energy-efficient AI acceleration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_05342 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | First Demonstration of Second-order Training of Deep Neural Networks with In-memory Analog Matrix Computing Zhang, Saitao Luo, Yubiao Wang, Shiqing Zuo, Pushen Li, Yongxiang Pan, Lunshuai Miao, Zheng Sun, Zhong Emerging Technologies Hardware Architecture Neural and Evolutionary Computing Second-order optimization methods, which leverage curvature information, offer faster and more stable convergence than first-order methods such as stochastic gradient descent (SGD) and Adam. However, their practical adoption is hindered by the prohibitively high cost of inverting the second-order information matrix, particularly in large-scale neural network training. Here, we present the first demonstration of a second-order optimizer powered by in-memory analog matrix computing (AMC) using resistive random-access memory (RRAM), which performs matrix inversion (INV) in a single step. We validate the optimizer by training a two-layer convolutional neural network (CNN) for handwritten letter classification, achieving 26% and 61% fewer training epochs than SGD with momentum and Adam, respectively. On a larger task using the same second-order method, our system delivers a 5.88x improvement in throughput and a 6.9x gain in energy efficiency compared to state-of-the-art digital processors. These results demonstrate the feasibility and effectiveness of AMC circuits for second-order neural network training, opening a new path toward energy-efficient AI acceleration. |
| title | First Demonstration of Second-order Training of Deep Neural Networks with In-memory Analog Matrix Computing |
| topic | Emerging Technologies Hardware Architecture Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2512.05342 |