HYDRA: Unifying Multi-modal Generation and Understanding via Representation-Harmonized Tokenization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912970649370624 |
|---|---|
| author | Qiu, Xuerui Cui, Yutao Zhang, Guozhen Li, Junzhe Hu, JiaKui Zhang, Xiao Li, Yang Liu, Songtao Yang, Miles Shi, Yu Zhong, Zhao Bo, Liefeng |
| author_facet | Qiu, Xuerui Cui, Yutao Zhang, Guozhen Li, Junzhe Hu, JiaKui Zhang, Xiao Li, Yang Liu, Songtao Yang, Miles Shi, Yu Zhong, Zhao Bo, Liefeng |
| contents | Unified Multimodal Models struggle to bridge the fundamental gap between the abstract representations needed for visual understanding and the detailed primitives required for generation. Existing approaches typically compromise by employing decoupled encoders, stacking representation encoder atop VAEs, or utilizing discrete quantization. However, these methods often disrupt information coherence and lead to optimization conflicts. To this end, we introduce HYDRA-TOK, a representation-harmonized pure ViT in the insight that visual modeling should evolve from generation to understanding. HYDRA-TOK reformulates the standard backbone into a progressive learner that transitions from a Gen-ViT, which captures structure-preserving primitives, to a Sem-ViT for semantic encoding. Crucially, this transition is mediated by a Generation-Semantic Bottleneck (GSB), which compresses features into a low-dimensional space to filter noise for robust synthesis, then restores dimensionality to empower complex semantic comprehension. Built upon this foundation, we present HYDRA, a native unified framework integrating perception and generation within a single parameter space. Extensive experiments establish HYDRA as a new state-of-the-art. It sets a benchmark in visual reconstruction (rFID 0.08) and achieves top-tier generation performance on GenEval (0.86), DPG-Bench (86.4), and WISE (0.53), while simultaneously outperforming previous native UMMs by an average of 10.0 points across eight challenging understanding benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_15228 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HYDRA: Unifying Multi-modal Generation and Understanding via Representation-Harmonized Tokenization Qiu, Xuerui Cui, Yutao Zhang, Guozhen Li, Junzhe Hu, JiaKui Zhang, Xiao Li, Yang Liu, Songtao Yang, Miles Shi, Yu Zhong, Zhao Bo, Liefeng Computer Vision and Pattern Recognition Unified Multimodal Models struggle to bridge the fundamental gap between the abstract representations needed for visual understanding and the detailed primitives required for generation. Existing approaches typically compromise by employing decoupled encoders, stacking representation encoder atop VAEs, or utilizing discrete quantization. However, these methods often disrupt information coherence and lead to optimization conflicts. To this end, we introduce HYDRA-TOK, a representation-harmonized pure ViT in the insight that visual modeling should evolve from generation to understanding. HYDRA-TOK reformulates the standard backbone into a progressive learner that transitions from a Gen-ViT, which captures structure-preserving primitives, to a Sem-ViT for semantic encoding. Crucially, this transition is mediated by a Generation-Semantic Bottleneck (GSB), which compresses features into a low-dimensional space to filter noise for robust synthesis, then restores dimensionality to empower complex semantic comprehension. Built upon this foundation, we present HYDRA, a native unified framework integrating perception and generation within a single parameter space. Extensive experiments establish HYDRA as a new state-of-the-art. It sets a benchmark in visual reconstruction (rFID 0.08) and achieves top-tier generation performance on GenEval (0.86), DPG-Bench (86.4), and WISE (0.53), while simultaneously outperforming previous native UMMs by an average of 10.0 points across eight challenging understanding benchmarks. |
| title | HYDRA: Unifying Multi-modal Generation and Understanding via Representation-Harmonized Tokenization |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.15228 |