Characterizing and Efficiently Accelerating Multimodal Generation Model Inference
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909606165348352 |
|---|---|
| author | Lee, Yejin Sun, Anna Hosmer, Basil Acun, Bilge Balioglu, Can Wang, Changhan Hernandez, Charles David Puhrsch, Christian Haziza, Daniel Guessous, Driss Massa, Francisco Kahn, Jacob Wan, Jeffrey Reizenstein, Jeremy Zhai, Jiaqi Isaacson, Joe Schlosser, Joel Pino, Juan Sadagopan, Kaushik Ram Shamis, Leonid Ma, Linjian Hwang, Min-Jae Chen, Mingda Elhoushi, Mostafa Rodriguez, Pedro Pasunuru, Ram Yih, Scott Popuri, Sravya Liu, Xing Wu, Carole-Jean |
| author_facet | Lee, Yejin Sun, Anna Hosmer, Basil Acun, Bilge Balioglu, Can Wang, Changhan Hernandez, Charles David Puhrsch, Christian Haziza, Daniel Guessous, Driss Massa, Francisco Kahn, Jacob Wan, Jeffrey Reizenstein, Jeremy Zhai, Jiaqi Isaacson, Joe Schlosser, Joel Pino, Juan Sadagopan, Kaushik Ram Shamis, Leonid Ma, Linjian Hwang, Min-Jae Chen, Mingda Elhoushi, Mostafa Rodriguez, Pedro Pasunuru, Ram Yih, Scott Popuri, Sravya Liu, Xing Wu, Carole-Jean |
| contents | Generative artificial intelligence (AI) technology is revolutionizing the computing industry. Not only its applications have broadened to various sectors but also poses new system design and optimization opportunities. The technology is capable of understanding and responding in multiple modalities. However, the advanced capability currently comes with significant system resource demands. To sustainably scale generative AI capabilities to billions of users in the world, inference must be fast and efficient. This paper pinpoints key system design and optimization opportunities by characterizing a family of emerging multi-modal generation models on real systems. Auto-regressive token generation is a critical latency performance bottleneck, typically dominated by GPU idle time. In addition to memory-intensive attention across the generative AI models, linear operations constitute significant inference latency due to the feed forward networks in Transformer-based models. We demonstrate that state-of-the-art optimization levers, spanning from applications to system software and hardware, set a 3.88x better baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_00215 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Characterizing and Efficiently Accelerating Multimodal Generation Model Inference Lee, Yejin Sun, Anna Hosmer, Basil Acun, Bilge Balioglu, Can Wang, Changhan Hernandez, Charles David Puhrsch, Christian Haziza, Daniel Guessous, Driss Massa, Francisco Kahn, Jacob Wan, Jeffrey Reizenstein, Jeremy Zhai, Jiaqi Isaacson, Joe Schlosser, Joel Pino, Juan Sadagopan, Kaushik Ram Shamis, Leonid Ma, Linjian Hwang, Min-Jae Chen, Mingda Elhoushi, Mostafa Rodriguez, Pedro Pasunuru, Ram Yih, Scott Popuri, Sravya Liu, Xing Wu, Carole-Jean Machine Learning Generative artificial intelligence (AI) technology is revolutionizing the computing industry. Not only its applications have broadened to various sectors but also poses new system design and optimization opportunities. The technology is capable of understanding and responding in multiple modalities. However, the advanced capability currently comes with significant system resource demands. To sustainably scale generative AI capabilities to billions of users in the world, inference must be fast and efficient. This paper pinpoints key system design and optimization opportunities by characterizing a family of emerging multi-modal generation models on real systems. Auto-regressive token generation is a critical latency performance bottleneck, typically dominated by GPU idle time. In addition to memory-intensive attention across the generative AI models, linear operations constitute significant inference latency due to the feed forward networks in Transformer-based models. We demonstrate that state-of-the-art optimization levers, spanning from applications to system software and hardware, set a 3.88x better baseline. |
| title | Characterizing and Efficiently Accelerating Multimodal Generation Model Inference |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2410.00215 |