FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918195522174976 |
|---|---|
| author | Tanikanti, Aditya Côté, Benoit Guo, Yanfei Chen, Le Saint, Nickolaus Chard, Ryan Raffenetti, Ken Thakur, Rajeev Uram, Thomas Foster, Ian Papka, Michael E. Vishwanath, Venkatram |
| author_facet | Tanikanti, Aditya Côté, Benoit Guo, Yanfei Chen, Le Saint, Nickolaus Chard, Ryan Raffenetti, Ken Thakur, Rajeev Uram, Thomas Foster, Ian Papka, Michael E. Vishwanath, Venkatram |
| contents | We present the Federated Inference Resource Scheduling Toolkit (FIRST), a framework enabling Inference-as-a-Service across distributed High-Performance Computing (HPC) clusters. FIRST provides cloud-like access to diverse AI models, like Large Language Models (LLMs), on existing HPC infrastructure. Leveraging Globus Auth and Globus Compute, the system allows researchers to run parallel inference workloads via an OpenAI-compliant API on private, secure environments. This cluster-agnostic API allows requests to be distributed across federated clusters, targeting numerous hosted models. FIRST supports multiple inference backends (e.g., vLLM), auto-scales resources, maintains "hot" nodes for low-latency execution, and offers both high-throughput batch and interactive modes. The framework addresses the growing demand for private, secure, and scalable AI inference in scientific workflows, allowing researchers to generate billions of tokens daily on-premises without relying on commercial cloud infrastructure. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_13724 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access Tanikanti, Aditya Côté, Benoit Guo, Yanfei Chen, Le Saint, Nickolaus Chard, Ryan Raffenetti, Ken Thakur, Rajeev Uram, Thomas Foster, Ian Papka, Michael E. Vishwanath, Venkatram Distributed, Parallel, and Cluster Computing Artificial Intelligence Software Engineering We present the Federated Inference Resource Scheduling Toolkit (FIRST), a framework enabling Inference-as-a-Service across distributed High-Performance Computing (HPC) clusters. FIRST provides cloud-like access to diverse AI models, like Large Language Models (LLMs), on existing HPC infrastructure. Leveraging Globus Auth and Globus Compute, the system allows researchers to run parallel inference workloads via an OpenAI-compliant API on private, secure environments. This cluster-agnostic API allows requests to be distributed across federated clusters, targeting numerous hosted models. FIRST supports multiple inference backends (e.g., vLLM), auto-scales resources, maintains "hot" nodes for low-latency execution, and offers both high-throughput batch and interactive modes. The framework addresses the growing demand for private, secure, and scalable AI inference in scientific workflows, allowing researchers to generate billions of tokens daily on-premises without relying on commercial cloud infrastructure. |
| title | FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Access |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Software Engineering |
| url | https://arxiv.org/abs/2510.13724 |