Theia: Distilling Diverse Vision Foundation Models for Robot Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909343942705152 |
|---|---|
| author | Shang, Jinghuan Schmeckpeper, Karl May, Brandon B. Minniti, Maria Vittoria Kelestemur, Tarik Watkins, David Herlant, Laura |
| author_facet | Shang, Jinghuan Schmeckpeper, Karl May, Brandon B. Minniti, Maria Vittoria Kelestemur, Tarik Watkins, David Herlant, Laura |
| contents | Vision-based robot policy learning, which maps visual inputs to actions, necessitates a holistic understanding of diverse visual tasks beyond single-task needs like classification or segmentation. Inspired by this, we introduce Theia, a vision foundation model for robot learning that distills multiple off-the-shelf vision foundation models trained on varied vision tasks. Theia's rich visual representations encode diverse visual knowledge, enhancing downstream robot learning. Extensive experiments demonstrate that Theia outperforms its teacher models and prior robot learning models using less training data and smaller model sizes. Additionally, we quantify the quality of pre-trained visual representations and hypothesize that higher entropy in feature norm distributions leads to improved robot learning performance. Code, models, and demo are available at https://theia.theaiinstitute.com. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_20179 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Theia: Distilling Diverse Vision Foundation Models for Robot Learning Shang, Jinghuan Schmeckpeper, Karl May, Brandon B. Minniti, Maria Vittoria Kelestemur, Tarik Watkins, David Herlant, Laura Robotics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Vision-based robot policy learning, which maps visual inputs to actions, necessitates a holistic understanding of diverse visual tasks beyond single-task needs like classification or segmentation. Inspired by this, we introduce Theia, a vision foundation model for robot learning that distills multiple off-the-shelf vision foundation models trained on varied vision tasks. Theia's rich visual representations encode diverse visual knowledge, enhancing downstream robot learning. Extensive experiments demonstrate that Theia outperforms its teacher models and prior robot learning models using less training data and smaller model sizes. Additionally, we quantify the quality of pre-trained visual representations and hypothesize that higher entropy in feature norm distributions leads to improved robot learning performance. Code, models, and demo are available at https://theia.theaiinstitute.com. |
| title | Theia: Distilling Diverse Vision Foundation Models for Robot Learning |
| topic | Robotics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2407.20179 |