COSMIC: Enabling Full-Stack Co-Design and Optimization of Distributed Machine Learning Systems
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918027974410240 |
|---|---|
| author | Raju, Aditi Ni, Jared Won, William Man, Changhai Krishnan, Srivatsan Sridharan, Srinivas Yazdanbakhsh, Amir Krishna, Tushar Reddi, Vijay Janapa |
| author_facet | Raju, Aditi Ni, Jared Won, William Man, Changhai Krishnan, Srivatsan Sridharan, Srinivas Yazdanbakhsh, Amir Krishna, Tushar Reddi, Vijay Janapa |
| contents | Large-scale machine learning models necessitate distributed systems, posing significant design challenges due to the large parameter space across distinct design stacks. Existing studies often focus on optimizing individual system aspects in isolation. This work challenges this limitation and introduces COSMIC, a full-stack distributed machine learning systems environment enabling end-to-end simulation and agent-based design space exploration. To facilitate efficient exploration and optimization across the entire stack, we introduce Parameter Set Architecture-an abstraction concept analogous to the instruction set architecture-abstracting away configuration complexities of agent-based search methods. Case studies demonstrate COSMIC's ability to consolidate parameters across multiple layers of design abstraction, discovering eight non-obvious high-performance system configurations across four transformer-based models with up to 175 billion parameters. By optimizing across the stack, COSMIC full-stack optimization delivers 1.50-48.41x higher performance compared to the isolated single-stack optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15020 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | COSMIC: Enabling Full-Stack Co-Design and Optimization of Distributed Machine Learning Systems Raju, Aditi Ni, Jared Won, William Man, Changhai Krishnan, Srivatsan Sridharan, Srinivas Yazdanbakhsh, Amir Krishna, Tushar Reddi, Vijay Janapa Distributed, Parallel, and Cluster Computing Large-scale machine learning models necessitate distributed systems, posing significant design challenges due to the large parameter space across distinct design stacks. Existing studies often focus on optimizing individual system aspects in isolation. This work challenges this limitation and introduces COSMIC, a full-stack distributed machine learning systems environment enabling end-to-end simulation and agent-based design space exploration. To facilitate efficient exploration and optimization across the entire stack, we introduce Parameter Set Architecture-an abstraction concept analogous to the instruction set architecture-abstracting away configuration complexities of agent-based search methods. Case studies demonstrate COSMIC's ability to consolidate parameters across multiple layers of design abstraction, discovering eight non-obvious high-performance system configurations across four transformer-based models with up to 175 billion parameters. By optimizing across the stack, COSMIC full-stack optimization delivers 1.50-48.41x higher performance compared to the isolated single-stack optimization. |
| title | COSMIC: Enabling Full-Stack Co-Design and Optimization of Distributed Machine Learning Systems |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2505.15020 |