T5Gemma 2: Seeing, Reading, and Understanding Longer
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915692833406976 |
|---|---|
| author | Zhang, Biao Suganthan, Paul Liu, Gaël Philippov, Ilya Dua, Sahil Hora, Ben Black, Kat Martins, Gus Sanseviero, Omar Pathak, Shreya Hardin, Cassidy Visin, Francesco Zhang, Jiageng Kenealy, Kathleen Yin, Qin Song, Xiaodan Lacombe, Olivier Joulin, Armand Warkentin, Tris Roberts, Adam |
| author_facet | Zhang, Biao Suganthan, Paul Liu, Gaël Philippov, Ilya Dua, Sahil Hora, Ben Black, Kat Martins, Gus Sanseviero, Omar Pathak, Shreya Hardin, Cassidy Visin, Francesco Zhang, Jiageng Kenealy, Kathleen Yin, Qin Song, Xiaodan Lacombe, Olivier Joulin, Armand Warkentin, Tris Roberts, Adam |
| contents | We introduce T5Gemma 2, the next generation of the T5Gemma family of lightweight open encoder-decoder models, featuring strong multilingual, multimodal and long-context capabilities. T5Gemma 2 follows the adaptation recipe (via UL2) in T5Gemma -- adapting a pretrained decoder-only model into an encoder-decoder model, and extends it from text-only regime to multimodal based on the Gemma 3 models. We further propose two methods to improve the efficiency: tied word embedding that shares all embeddings across encoder and decoder, and merged attention that unifies decoder self- and cross-attention into a single joint module. Experiments demonstrate the generality of the adaptation strategy over architectures and modalities as well as the unique strength of the encoder-decoder architecture on long context modeling. Similar to T5Gemma, T5Gemma 2 yields comparable or better pretraining performance and significantly improved post-training performance than its Gemma 3 counterpart. We release the pretrained models (270M-270M, 1B-1B and 4B-4B) to the community for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_14856 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | T5Gemma 2: Seeing, Reading, and Understanding Longer Zhang, Biao Suganthan, Paul Liu, Gaël Philippov, Ilya Dua, Sahil Hora, Ben Black, Kat Martins, Gus Sanseviero, Omar Pathak, Shreya Hardin, Cassidy Visin, Francesco Zhang, Jiageng Kenealy, Kathleen Yin, Qin Song, Xiaodan Lacombe, Olivier Joulin, Armand Warkentin, Tris Roberts, Adam Computation and Language We introduce T5Gemma 2, the next generation of the T5Gemma family of lightweight open encoder-decoder models, featuring strong multilingual, multimodal and long-context capabilities. T5Gemma 2 follows the adaptation recipe (via UL2) in T5Gemma -- adapting a pretrained decoder-only model into an encoder-decoder model, and extends it from text-only regime to multimodal based on the Gemma 3 models. We further propose two methods to improve the efficiency: tied word embedding that shares all embeddings across encoder and decoder, and merged attention that unifies decoder self- and cross-attention into a single joint module. Experiments demonstrate the generality of the adaptation strategy over architectures and modalities as well as the unique strength of the encoder-decoder architecture on long context modeling. Similar to T5Gemma, T5Gemma 2 yields comparable or better pretraining performance and significantly improved post-training performance than its Gemma 3 counterpart. We release the pretrained models (270M-270M, 1B-1B and 4B-4B) to the community for future research. |
| title | T5Gemma 2: Seeing, Reading, and Understanding Longer |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.14856 |