Neuromorphic spatiotemporal optical flow: Enabling ultrafast visual perception beyond human capabilities
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917906332254208 |
|---|---|
| author | Wang, Shengbo Zhao, Jingwen Pu, Tongming Zhao, Liangbing Guo, Xiaoyu Cheng, Yue Li, Cong Ma, Weihao Tang, Chenyu Xu, Zhenyu Wang, Ningli Occhipinti, Luigi Nathan, Arokia Dahiya, Ravinder Wu, Huaqiang Tao, Li Gao, Shuo |
| author_facet | Wang, Shengbo Zhao, Jingwen Pu, Tongming Zhao, Liangbing Guo, Xiaoyu Cheng, Yue Li, Cong Ma, Weihao Tang, Chenyu Xu, Zhenyu Wang, Ningli Occhipinti, Luigi Nathan, Arokia Dahiya, Ravinder Wu, Huaqiang Tao, Li Gao, Shuo |
| contents | Optical flow, inspired by the mechanisms of biological visual systems, calculates spatial motion vectors within visual scenes that are necessary for enabling robotics to excel in complex and dynamic working environments. However, current optical flow algorithms, despite human-competitive task performance on benchmark datasets, remain constrained by unacceptable time delays (~0.6 seconds per inference, 4X human processing speed) in practical deployment. Here, we introduce a neuromorphic optical flow approach that addresses delay bottlenecks by encoding temporal information directly in a synaptic transistor array to assist spatial motion analysis. Compared to conventional spatial-only optical flow methods, our spatiotemporal neuromorphic optical flow offers the spatial-temporal consistency of motion information, rapidly identifying regions of interest in as little as 1-2 ms using the temporal motion cues derived from the embedded temporal information in the two-dimensional floating gate synaptic transistors. Thus, the visual input can be selectively filtered to achieve faster velocity calculations and various task execution. At the hardware level, due to the atomically sharp interfaces between distinct functional layers in two-dimensional van der Waals heterostructures, the synaptic transistor offers high-frequency response (~100 μs), robust non-volatility (>10000 s), and excellent endurance (>8000 cycles), enabling robust visual processing. In software benchmarks, our system outperforms state-of-the-art algorithms with a 400% speedup, frequently surpassing human-level performance while maintaining or enhancing accuracy by utilizing the temporal priors provided by the embedded temporal information. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_15345 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Neuromorphic spatiotemporal optical flow: Enabling ultrafast visual perception beyond human capabilities Wang, Shengbo Zhao, Jingwen Pu, Tongming Zhao, Liangbing Guo, Xiaoyu Cheng, Yue Li, Cong Ma, Weihao Tang, Chenyu Xu, Zhenyu Wang, Ningli Occhipinti, Luigi Nathan, Arokia Dahiya, Ravinder Wu, Huaqiang Tao, Li Gao, Shuo Computer Vision and Pattern Recognition Robotics Optical flow, inspired by the mechanisms of biological visual systems, calculates spatial motion vectors within visual scenes that are necessary for enabling robotics to excel in complex and dynamic working environments. However, current optical flow algorithms, despite human-competitive task performance on benchmark datasets, remain constrained by unacceptable time delays (~0.6 seconds per inference, 4X human processing speed) in practical deployment. Here, we introduce a neuromorphic optical flow approach that addresses delay bottlenecks by encoding temporal information directly in a synaptic transistor array to assist spatial motion analysis. Compared to conventional spatial-only optical flow methods, our spatiotemporal neuromorphic optical flow offers the spatial-temporal consistency of motion information, rapidly identifying regions of interest in as little as 1-2 ms using the temporal motion cues derived from the embedded temporal information in the two-dimensional floating gate synaptic transistors. Thus, the visual input can be selectively filtered to achieve faster velocity calculations and various task execution. At the hardware level, due to the atomically sharp interfaces between distinct functional layers in two-dimensional van der Waals heterostructures, the synaptic transistor offers high-frequency response (~100 μs), robust non-volatility (>10000 s), and excellent endurance (>8000 cycles), enabling robust visual processing. In software benchmarks, our system outperforms state-of-the-art algorithms with a 400% speedup, frequently surpassing human-level performance while maintaining or enhancing accuracy by utilizing the temporal priors provided by the embedded temporal information. |
| title | Neuromorphic spatiotemporal optical flow: Enabling ultrafast visual perception beyond human capabilities |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2409.15345 |