SkyMemory: A LEO Edge Cache for Transformer Inference Optimization and Scale Out
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909617500454912 |
|---|---|
| author | Sandholm, Thomas Mukherjee, Sayandev Cheng, Lin Huberman, Bernardo A. |
| author_facet | Sandholm, Thomas Mukherjee, Sayandev Cheng, Lin Huberman, Bernardo A. |
| contents | We expand the scope of cache memory to include LEO constellations, which are highly distributed systems with thousands of satellites connected with free-space optics inter-satellite links (ISL) always only one hop from any point on earth. We show how to increase the number of cache hits and improve the speed of inference for the important use case of LLMs. These benefits apply not only to LLMs, both terrestrially hosted and on satellites, but also generalize to any cache distributed over multiple locations that needs to be accessed in a timely manner. We show the benefit of our key value cache (KVC) protocol in simulations and present a proof-of-concept implementation of the protocol for KVCs on a testbed comprising 5 Intel NUC Linux mini PCs hosting a 19x5 constellation, with an NVIDIA Jetson Nano 8GB GPU hosting the LLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14427 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SkyMemory: A LEO Edge Cache for Transformer Inference Optimization and Scale Out Sandholm, Thomas Mukherjee, Sayandev Cheng, Lin Huberman, Bernardo A. Distributed, Parallel, and Cluster Computing We expand the scope of cache memory to include LEO constellations, which are highly distributed systems with thousands of satellites connected with free-space optics inter-satellite links (ISL) always only one hop from any point on earth. We show how to increase the number of cache hits and improve the speed of inference for the important use case of LLMs. These benefits apply not only to LLMs, both terrestrially hosted and on satellites, but also generalize to any cache distributed over multiple locations that needs to be accessed in a timely manner. We show the benefit of our key value cache (KVC) protocol in simulations and present a proof-of-concept implementation of the protocol for KVCs on a testbed comprising 5 Intel NUC Linux mini PCs hosting a 19x5 constellation, with an NVIDIA Jetson Nano 8GB GPU hosting the LLM. |
| title | SkyMemory: A LEO Edge Cache for Transformer Inference Optimization and Scale Out |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2505.14427 |