OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911681388478464 |
|---|---|
| author | Ding, Yue Ji, Yiyan Li, Jungang Liu, Xuyang Chen, Xinlong Wu, Junfei Li, Bozhou Zeng, Bohan Shi, Yang Guan, Yushuo Zhang, Yuanxing Liu, Jiaheng Liu, Qiang Wan, Pengfei Wang, Liang |
| author_facet | Ding, Yue Ji, Yiyan Li, Jungang Liu, Xuyang Chen, Xinlong Wu, Junfei Li, Bozhou Zeng, Bohan Shi, Yang Guan, Yushuo Zhang, Yuanxing Liu, Jiaheng Liu, Qiang Wan, Pengfei Wang, Liang |
| contents | Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks demonstrate the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT introduces only 4.85M parameters while maintaining lower latency than training-free baselines such as OmniZip. With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_04804 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models Ding, Yue Ji, Yiyan Li, Jungang Liu, Xuyang Chen, Xinlong Wu, Junfei Li, Bozhou Zeng, Bohan Shi, Yang Guan, Yushuo Zhang, Yuanxing Liu, Jiaheng Liu, Qiang Wan, Pengfei Wang, Liang Computation and Language Omni-modal Large Language Models (Omni-LLMs) have demonstrated strong capabilities in audio-video understanding tasks. However, their reliance on long multimodal token sequences leads to substantial computational overhead. Despite this challenge, token compression methods designed for Omni-LLMs remain limited. To bridge this gap, we propose OmniSIFT (Omni-modal Spatio-temporal Informed Fine-grained Token compression), a modality-asymmetric token compression framework tailored for Omni-LLMs. Specifically, OmniSIFT adopts a two-stage compression strategy: (i) a spatio-temporal video pruning module that removes video redundancy arising from both intra-frame structure and inter-frame overlap, and (ii) a vision-guided audio selection module that filters audio tokens. The entire framework is optimized end-to-end via a differentiable straight-through estimator. Extensive experiments on five representative benchmarks demonstrate the efficacy and robustness of OmniSIFT. Notably, for Qwen2.5-Omni-7B, OmniSIFT introduces only 4.85M parameters while maintaining lower latency than training-free baselines such as OmniZip. With merely 25% of the original token context, OmniSIFT consistently outperforms all compression baselines and even surpasses the performance of the full-token model on several tasks. |
| title | OmniSIFT: Modality-Asymmetric Token Compression for Efficient Omni-modal Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.04804 |