Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.15950 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908665950240768 |
|---|---|
| author | DeBole, Michael V. Appuswamy, Rathinakumar McGlohon, Neil Taba, Brian Esser, Steven K. Akopyan, Filipp Arthur, John V. Amir, Arnon Andreopoulos, Alexander Carlson, Peter J. Cassidy, Andrew S. Datta, Pallab Flickner, Myron D. Gandhasri, Rajamohan Garreau, Guillaume J. Ito, Megumi Klamo, Jennifer L. Kusnitz, Jeffrey A. McClatchey, Nathaniel J. McKinstry, Jeffrey L. Nayak, Tapan K. Otero, Carlos Ortega Penner, Hartmut Risk, William P. Sawada, Jun Sivagnaname, Jay Smith, Daniel F. Sousa, Rafael Terrizzano, Ignacio Ueda, Takanori Gray-Donald, Trent Cox, David Modha, Dharmendra S. |
| author_facet | DeBole, Michael V. Appuswamy, Rathinakumar McGlohon, Neil Taba, Brian Esser, Steven K. Akopyan, Filipp Arthur, John V. Amir, Arnon Andreopoulos, Alexander Carlson, Peter J. Cassidy, Andrew S. Datta, Pallab Flickner, Myron D. Gandhasri, Rajamohan Garreau, Guillaume J. Ito, Megumi Klamo, Jennifer L. Kusnitz, Jeffrey A. McClatchey, Nathaniel J. McKinstry, Jeffrey L. Nayak, Tapan K. Otero, Carlos Ortega Penner, Hartmut Risk, William P. Sawada, Jun Sivagnaname, Jay Smith, Daniel F. Sousa, Rafael Terrizzano, Ignacio Ueda, Takanori Gray-Donald, Trent Cox, David Modha, Dharmendra S. |
| contents | A vertically integrated, end-to-end, research prototype system combines 288 NorthPole neural inference accelerator cards, offline training algorithms, a high-performance runtime stack, and a containerized inference pipeline to deliver a scalable and efficient cloud inference service. The system delivers 115 peta-ops at 4-bit integer precision and 3.7 PB/s of memory bandwidth across 18 2U servers, while consuming only 30 kW of power and weighing 730 kg in a 0.67 m^2 42U rack footprint. The system can run 3 simultaneous instances of the 8-billion-parameter open-source IBM Granite-3.3-8b-instruct model at 2,048 context length with 28 simultaneous users and a per-user inter-token latency of 2.8 ms. The system is scalable, modular, and reconfigurable, supporting various model sizes and context lengths, and is ideal for deploying agentic workflows for enterprise AI applications in existing data center (cloud, on-prem) environments. For example, the system can support 18 instances of a 3-billion-parameter model or a single instance of a 70-billion-parameter model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_15950 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Scalable NorthPole System with End-to-End Vertical Integration for Low-Latency and Energy-Efficient LLM Inference DeBole, Michael V. Appuswamy, Rathinakumar McGlohon, Neil Taba, Brian Esser, Steven K. Akopyan, Filipp Arthur, John V. Amir, Arnon Andreopoulos, Alexander Carlson, Peter J. Cassidy, Andrew S. Datta, Pallab Flickner, Myron D. Gandhasri, Rajamohan Garreau, Guillaume J. Ito, Megumi Klamo, Jennifer L. Kusnitz, Jeffrey A. McClatchey, Nathaniel J. McKinstry, Jeffrey L. Nayak, Tapan K. Otero, Carlos Ortega Penner, Hartmut Risk, William P. Sawada, Jun Sivagnaname, Jay Smith, Daniel F. Sousa, Rafael Terrizzano, Ignacio Ueda, Takanori Gray-Donald, Trent Cox, David Modha, Dharmendra S. Distributed, Parallel, and Cluster Computing Artificial Intelligence Hardware Architecture A vertically integrated, end-to-end, research prototype system combines 288 NorthPole neural inference accelerator cards, offline training algorithms, a high-performance runtime stack, and a containerized inference pipeline to deliver a scalable and efficient cloud inference service. The system delivers 115 peta-ops at 4-bit integer precision and 3.7 PB/s of memory bandwidth across 18 2U servers, while consuming only 30 kW of power and weighing 730 kg in a 0.67 m^2 42U rack footprint. The system can run 3 simultaneous instances of the 8-billion-parameter open-source IBM Granite-3.3-8b-instruct model at 2,048 context length with 28 simultaneous users and a per-user inter-token latency of 2.8 ms. The system is scalable, modular, and reconfigurable, supporting various model sizes and context lengths, and is ideal for deploying agentic workflows for enterprise AI applications in existing data center (cloud, on-prem) environments. For example, the system can support 18 instances of a 3-billion-parameter model or a single instance of a 70-billion-parameter model. |
| title | A Scalable NorthPole System with End-to-End Vertical Integration for Low-Latency and Energy-Efficient LLM Inference |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Hardware Architecture |
| url | https://arxiv.org/abs/2511.15950 |