GATS: Gather-Attend-Scatter
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929211007041536 |
|---|---|
| author | Zolna, Konrad Cabi, Serkan Chen, Yutian Lau, Eric Fantacci, Claudio Pasukonis, Jurgis Springenberg, Jost Tobias Colmenarejo, Sergio Gomez |
| author_facet | Zolna, Konrad Cabi, Serkan Chen, Yutian Lau, Eric Fantacci, Claudio Pasukonis, Jurgis Springenberg, Jost Tobias Colmenarejo, Sergio Gomez |
| contents | As the AI community increasingly adopts large-scale models, it is crucial to develop general and flexible tools to integrate them. We introduce Gather-Attend-Scatter (GATS), a novel module that enables seamless combination of pretrained foundation models, both trainable and frozen, into larger multimodal networks. GATS empowers AI systems to process and generate information across multiple modalities at different rates. In contrast to traditional fine-tuning, GATS allows for the original component models to remain frozen, avoiding the risk of them losing important knowledge acquired during the pretraining phase. We demonstrate the utility and versatility of GATS with a few experiments across games, robotics, and multimodal input-output systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_08525 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GATS: Gather-Attend-Scatter Zolna, Konrad Cabi, Serkan Chen, Yutian Lau, Eric Fantacci, Claudio Pasukonis, Jurgis Springenberg, Jost Tobias Colmenarejo, Sergio Gomez Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Robotics As the AI community increasingly adopts large-scale models, it is crucial to develop general and flexible tools to integrate them. We introduce Gather-Attend-Scatter (GATS), a novel module that enables seamless combination of pretrained foundation models, both trainable and frozen, into larger multimodal networks. GATS empowers AI systems to process and generate information across multiple modalities at different rates. In contrast to traditional fine-tuning, GATS allows for the original component models to remain frozen, avoiding the risk of them losing important knowledge acquired during the pretraining phase. We demonstrate the utility and versatility of GATS with a few experiments across games, robotics, and multimodal input-output systems. |
| title | GATS: Gather-Attend-Scatter |
| topic | Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Robotics |
| url | https://arxiv.org/abs/2401.08525 |