JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911187873038336 |
|---|---|
| author | Wan, Siheng Yao, Zhengtao Li, Zhengdao Dong, Junhao Li, Yanshu Li, Yikai Li, Linshan Xu, Haoyan Li, Yijiang Dong, Zhikang Wang, Huacan Shen, Jifeng |
| author_facet | Wan, Siheng Yao, Zhengtao Li, Zhengdao Dong, Junhao Li, Yanshu Li, Yikai Li, Linshan Xu, Haoyan Li, Yijiang Dong, Zhikang Wang, Huacan Shen, Jifeng |
| contents | Modern Text-to-Image (T2I) generation increasingly relies on token-centric architectures that are trained with self-supervision, yet effectively fusing text with visual tokens remains a challenge. We propose \textbf{JEPA-T}, a unified multimodal framework that encodes images and captions into discrete visual and textual tokens, processed by a joint-embedding predictive Transformer. To enhance fusion, we incorporate cross-attention after the feature predictor for conditional denoising while maintaining a task-agnostic backbone. Additionally, raw texts embeddings are injected prior to the flow matching loss to improve alignment during training. During inference, the same network performs both class-conditional and free-text image generation by iteratively denoising visual tokens conditioned on text. Evaluations on ImageNet-1K demonstrate that JEPA-T achieves strong data efficiency, open-vocabulary generalization, and consistently outperforms non-fusion and late-fusion baselines. Our approach shows that late architectural fusion combined with objective-level alignment offers an effective balance between conditioning strength and backbone generality in token-based T2I.The code is now available: https://github.com/justin-herry/JEPA-T.git |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00974 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generation Wan, Siheng Yao, Zhengtao Li, Zhengdao Dong, Junhao Li, Yanshu Li, Yikai Li, Linshan Xu, Haoyan Li, Yijiang Dong, Zhikang Wang, Huacan Shen, Jifeng Computer Vision and Pattern Recognition Modern Text-to-Image (T2I) generation increasingly relies on token-centric architectures that are trained with self-supervision, yet effectively fusing text with visual tokens remains a challenge. We propose \textbf{JEPA-T}, a unified multimodal framework that encodes images and captions into discrete visual and textual tokens, processed by a joint-embedding predictive Transformer. To enhance fusion, we incorporate cross-attention after the feature predictor for conditional denoising while maintaining a task-agnostic backbone. Additionally, raw texts embeddings are injected prior to the flow matching loss to improve alignment during training. During inference, the same network performs both class-conditional and free-text image generation by iteratively denoising visual tokens conditioned on text. Evaluations on ImageNet-1K demonstrate that JEPA-T achieves strong data efficiency, open-vocabulary generalization, and consistently outperforms non-fusion and late-fusion baselines. Our approach shows that late architectural fusion combined with objective-level alignment offers an effective balance between conditioning strength and backbone generality in token-based T2I.The code is now available: https://github.com/justin-herry/JEPA-T.git |
| title | JEPA-T: Joint-Embedding Predictive Architecture with Text Fusion for Image Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2510.00974 |