Focus Session: Hardware and Software Techniques for Accelerating Multimodal Foundation Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911619551854592 |
|---|---|
| author | Shafique, Muhammad Basit, Abdul Hanif, Muhammad Abdullah Marchisio, Alberto Putra, Rachmad Vidya Wicaksana Shao, Minghao |
| author_facet | Shafique, Muhammad Basit, Abdul Hanif, Muhammad Abdullah Marchisio, Alberto Putra, Rachmad Vidya Wicaksana Shao, Minghao |
| contents | This work presents a multi-layered methodology for efficiently accelerating multimodal foundation models (MFMs). It combines hardware and software co-design of transformer blocks with an optimization pipeline that reduces computational and memory requirements. During model development, it employs performance enhancements through fine-tuning for domain-specific adaptation. Our methodology further incorporates hardware and software techniques for optimizing MFMs. Specifically, it employs MFM compression using hierarchy-aware mixed-precision quantization and structural pruning for transformer blocks and MLP channels. It also optimizes operations through speculative decoding, model cascading that routes queries through a small-to-large cascade and uses lightweight self-tests to determine when to escalate to larger models, as well as co-optimization of sequence length, visual resolution & stride, and graph-level operator fusion. To efficiently execute the model, the processing dataflow is optimized based on the underlying hardware architecture together with memory-efficient attention to meet on-chip bandwidth and latency budgets. To support this, a specialized hardware accelerator for the transformer workloads is employed, which can be developed through expert design or an LLM-aided design approach. We demonstrate the effectiveness of the proposed methodology on medical-MFMs and on code generation tasks, and conclude with extensions toward energy-efficient spiking-MFMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_21952 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Focus Session: Hardware and Software Techniques for Accelerating Multimodal Foundation Models Shafique, Muhammad Basit, Abdul Hanif, Muhammad Abdullah Marchisio, Alberto Putra, Rachmad Vidya Wicaksana Shao, Minghao Machine Learning Artificial Intelligence Hardware Architecture Neural and Evolutionary Computing Robotics This work presents a multi-layered methodology for efficiently accelerating multimodal foundation models (MFMs). It combines hardware and software co-design of transformer blocks with an optimization pipeline that reduces computational and memory requirements. During model development, it employs performance enhancements through fine-tuning for domain-specific adaptation. Our methodology further incorporates hardware and software techniques for optimizing MFMs. Specifically, it employs MFM compression using hierarchy-aware mixed-precision quantization and structural pruning for transformer blocks and MLP channels. It also optimizes operations through speculative decoding, model cascading that routes queries through a small-to-large cascade and uses lightweight self-tests to determine when to escalate to larger models, as well as co-optimization of sequence length, visual resolution & stride, and graph-level operator fusion. To efficiently execute the model, the processing dataflow is optimized based on the underlying hardware architecture together with memory-efficient attention to meet on-chip bandwidth and latency budgets. To support this, a specialized hardware accelerator for the transformer workloads is employed, which can be developed through expert design or an LLM-aided design approach. We demonstrate the effectiveness of the proposed methodology on medical-MFMs and on code generation tasks, and conclude with extensions toward energy-efficient spiking-MFMs. |
| title | Focus Session: Hardware and Software Techniques for Accelerating Multimodal Foundation Models |
| topic | Machine Learning Artificial Intelligence Hardware Architecture Neural and Evolutionary Computing Robotics |
| url | https://arxiv.org/abs/2604.21952 |