Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916070674137088 |
|---|---|
| author | Jiang, Yitong Wang, Hongjun McCarthy, Collin Ye, Hanrong Wehr, David Li, Xinhao Dou, Qi Xue, Tianfan Cheung, Ka Chun See, Simon Byeon, Wonmin Chen, Ke Han, Kai Gu, Jinwei Yin, Hongxu Molchanov, Pavlo Kautz, Jan Liu, Sifei |
| author_facet | Jiang, Yitong Wang, Hongjun McCarthy, Collin Ye, Hanrong Wehr, David Li, Xinhao Dou, Qi Xue, Tianfan Cheung, Ka Chun See, Simon Byeon, Wonmin Chen, Ke Han, Kai Gu, Jinwei Yin, Hongxu Molchanov, Pavlo Kautz, Jan Liu, Sifei |
| contents | Vision foundation models are bottlenecked by the quadratic cost of self-attention, which limits usable resolution and increases the cost of large-scale pretraining. Subquadratic alternatives such as linear attention and state-space models reduce this cost, but often serialize images into 1D token streams and weaken the 2D spatial structure important for vision. Generalized Spatial Propagation Networks (GSPN) instead propagate context directly on the 2D grid through line-scan recurrences, achieving near-linear complexity without positional embeddings, but have seen little use as foundation-scale encoders.
We present C-GSPN, a foundation-scale vision encoder based on 2D spatial propagation. C-GSPN makes the operator practical through three improvements: (1) a fast GSPN CUDA kernel that fuses per-step launches into a single warp-specialized implementation with shared-memory tiling, coalesced access, and a compact multi-channel propagation, reaching over 90% of peak memory bandwidth and running up to 40--52x faster than the original GSPN implementation; (2) a compressed latent-space propagation block with fused normalization, which turns kernel-level speed into block- and model-level efficiency; and (3) a two-stage cross-operator distillation recipe that trains the new architecture from an attention teacher without the cost of from-scratch foundation-scale training. Distilled with 600M image-text pairs, C-GSPN matches an isomorphic ViT baseline with 15% fewer parameters, improves ADE20K segmentation by +2.1%, transfers to high resolution with a fraction of the data needed from scratch, and delivers a 4x end-to-end block speedup at 2K with single-pass, tiling-free inference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_00746 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders Jiang, Yitong Wang, Hongjun McCarthy, Collin Ye, Hanrong Wehr, David Li, Xinhao Dou, Qi Xue, Tianfan Cheung, Ka Chun See, Simon Byeon, Wonmin Chen, Ke Han, Kai Gu, Jinwei Yin, Hongxu Molchanov, Pavlo Kautz, Jan Liu, Sifei Computer Vision and Pattern Recognition Machine Learning Vision foundation models are bottlenecked by the quadratic cost of self-attention, which limits usable resolution and increases the cost of large-scale pretraining. Subquadratic alternatives such as linear attention and state-space models reduce this cost, but often serialize images into 1D token streams and weaken the 2D spatial structure important for vision. Generalized Spatial Propagation Networks (GSPN) instead propagate context directly on the 2D grid through line-scan recurrences, achieving near-linear complexity without positional embeddings, but have seen little use as foundation-scale encoders. We present C-GSPN, a foundation-scale vision encoder based on 2D spatial propagation. C-GSPN makes the operator practical through three improvements: (1) a fast GSPN CUDA kernel that fuses per-step launches into a single warp-specialized implementation with shared-memory tiling, coalesced access, and a compact multi-channel propagation, reaching over 90% of peak memory bandwidth and running up to 40--52x faster than the original GSPN implementation; (2) a compressed latent-space propagation block with fused normalization, which turns kernel-level speed into block- and model-level efficiency; and (3) a two-stage cross-operator distillation recipe that trains the new architecture from an attention teacher without the cost of from-scratch foundation-scale training. Distilled with 600M image-text pairs, C-GSPN matches an isomorphic ViT baseline with 15% fewer parameters, improves ADE20K segmentation by +2.1%, transfers to high resolution with a fraction of the data needed from scratch, and delivers a 4x end-to-end block speedup at 2K with single-pass, tiling-free inference. |
| title | Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2606.00746 |