Language-Embedded Gaussian Splats (LEGS): Incrementally Building Room-Scale Representations with a Mobile Robot
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910621529800704 |
|---|---|
| author | Yu, Justin Hari, Kush Srinivas, Kishore El-Refai, Karim Rashid, Adam Kim, Chung Min Kerr, Justin Cheng, Richard Irshad, Muhammad Zubair Balakrishna, Ashwin Kollar, Thomas Goldberg, Ken |
| author_facet | Yu, Justin Hari, Kush Srinivas, Kishore El-Refai, Karim Rashid, Adam Kim, Chung Min Kerr, Justin Cheng, Richard Irshad, Muhammad Zubair Balakrishna, Ashwin Kollar, Thomas Goldberg, Ken |
| contents | Building semantic 3D maps is valuable for searching for objects of interest in offices, warehouses, stores, and homes. We present a mapping system that incrementally builds a Language-Embedded Gaussian Splat (LEGS): a detailed 3D scene representation that encodes both appearance and semantics in a unified representation. LEGS is trained online as a robot traverses its environment to enable localization of open-vocabulary object queries. We evaluate LEGS on 4 room-scale scenes where we query for objects in the scene to assess how LEGS can capture semantic meaning. We compare LEGS to LERF and find that while both systems have comparable object query success rates, LEGS trains over 3.5x faster than LERF. Results suggest that a multi-camera setup and incremental bundle adjustment can boost visual reconstruction quality in constrained robot trajectories, and suggest LEGS can localize open-vocabulary and long-tail object queries with up to 66% accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18108 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Language-Embedded Gaussian Splats (LEGS): Incrementally Building Room-Scale Representations with a Mobile Robot Yu, Justin Hari, Kush Srinivas, Kishore El-Refai, Karim Rashid, Adam Kim, Chung Min Kerr, Justin Cheng, Richard Irshad, Muhammad Zubair Balakrishna, Ashwin Kollar, Thomas Goldberg, Ken Robotics Building semantic 3D maps is valuable for searching for objects of interest in offices, warehouses, stores, and homes. We present a mapping system that incrementally builds a Language-Embedded Gaussian Splat (LEGS): a detailed 3D scene representation that encodes both appearance and semantics in a unified representation. LEGS is trained online as a robot traverses its environment to enable localization of open-vocabulary object queries. We evaluate LEGS on 4 room-scale scenes where we query for objects in the scene to assess how LEGS can capture semantic meaning. We compare LEGS to LERF and find that while both systems have comparable object query success rates, LEGS trains over 3.5x faster than LERF. Results suggest that a multi-camera setup and incremental bundle adjustment can boost visual reconstruction quality in constrained robot trajectories, and suggest LEGS can localize open-vocabulary and long-tail object queries with up to 66% accuracy. |
| title | Language-Embedded Gaussian Splats (LEGS): Incrementally Building Room-Scale Representations with a Mobile Robot |
| topic | Robotics |
| url | https://arxiv.org/abs/2409.18108 |