SWAN: World-Aware Adaptive Multimodal Networks for Runtime Variations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909006419722240 |
|---|---|
| author | Wu, Jason Jin, Shir-Kang Scott Yuan, Yuyang Wigness, Maggie Kaplan, Lance M. Qiu, Hang Srivastava, Mani |
| author_facet | Wu, Jason Jin, Shir-Kang Scott Yuan, Yuyang Wigness, Maggie Kaplan, Lance M. Qiu, Hang Srivastava, Mani |
| contents | Multimodal deep neural networks deployed in realistic environments must contend with runtime variations: changes in modality quality, overall input complexity, and available platform resources. Current networks struggle with such fluctuations -- adaptive networks cannot adhere to a strict compute budget, controller-based networks neglect to consider input complexity, and statically provisioned networks fail at all the above. Consequently, they do not extract maximum utility from the expended computational resources. We present SWAN (Sample and World-Aware Multimodal Network), the first adaptive multimodal network that accomplishes all three goals. SWAN employs a quality-aware controller to assign resources among modalities according to a variable user-specified maximum budget. Within this budget, an adaptive gating module further optimizes efficiency by scaling layer utilization according to sample complexity. For further gains, SWAN also employs a token dropping module that masks semantically irrelevant multimodal features before performing detections. We evaluate SWAN in the domain of autonomous driving with complex multi-object 3D detection, reducing FLOPs by up to 49% with minimal degradation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_26181 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SWAN: World-Aware Adaptive Multimodal Networks for Runtime Variations Wu, Jason Jin, Shir-Kang Scott Yuan, Yuyang Wigness, Maggie Kaplan, Lance M. Qiu, Hang Srivastava, Mani Machine Learning Multimodal deep neural networks deployed in realistic environments must contend with runtime variations: changes in modality quality, overall input complexity, and available platform resources. Current networks struggle with such fluctuations -- adaptive networks cannot adhere to a strict compute budget, controller-based networks neglect to consider input complexity, and statically provisioned networks fail at all the above. Consequently, they do not extract maximum utility from the expended computational resources. We present SWAN (Sample and World-Aware Multimodal Network), the first adaptive multimodal network that accomplishes all three goals. SWAN employs a quality-aware controller to assign resources among modalities according to a variable user-specified maximum budget. Within this budget, an adaptive gating module further optimizes efficiency by scaling layer utilization according to sample complexity. For further gains, SWAN also employs a token dropping module that masks semantically irrelevant multimodal features before performing detections. We evaluate SWAN in the domain of autonomous driving with complex multi-object 3D detection, reducing FLOPs by up to 49% with minimal degradation. |
| title | SWAN: World-Aware Adaptive Multimodal Networks for Runtime Variations |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2604.26181 |