Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.19561 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916414332338176 |
|---|---|
| author | Ren, Lianhai Li, Qianxiao |
| author_facet | Ren, Lianhai Li, Qianxiao |
| contents | We introduce a Model Predictive Control (MPC) framework for training deep neural networks, systematically unifying the Back-Propagation (BP) and Forward-Forward (FF) algorithms. At the same time, it gives rise to a range of intermediate training algorithms with varying look-forward horizons, leading to a performance-efficiency trade-off. We perform a precise analysis of this trade-off on a deep linear network, where the qualitative conclusions carry over to general networks. Based on our analysis, we propose a principled method to choose the optimization horizon based on given objectives and model specifications. Numerical results on various models and tasks demonstrate the versatility of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_19561 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unifying back-propagation and forward-forward algorithms through model predictive control Ren, Lianhai Li, Qianxiao Machine Learning Optimization and Control We introduce a Model Predictive Control (MPC) framework for training deep neural networks, systematically unifying the Back-Propagation (BP) and Forward-Forward (FF) algorithms. At the same time, it gives rise to a range of intermediate training algorithms with varying look-forward horizons, leading to a performance-efficiency trade-off. We perform a precise analysis of this trade-off on a deep linear network, where the qualitative conclusions carry over to general networks. Based on our analysis, we propose a principled method to choose the optimization horizon based on given objectives and model specifications. Numerical results on various models and tasks demonstrate the versatility of our method. |
| title | Unifying back-propagation and forward-forward algorithms through model predictive control |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2409.19561 |