LLM-Craft: Robotic Crafting of Elasto-Plastic Objects with Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910996866531328 |
|---|---|
| author | Bartsch, Alison Farimani, Amir Barati |
| author_facet | Bartsch, Alison Farimani, Amir Barati |
| contents | When humans create sculptures, we are able to reason about how geometrically we need to alter the clay state to reach our target goal. We are not computing point-wise similarity metrics, or reasoning about low-level positioning of our tools, but instead determining the higher-level changes that need to be made. In this work, we propose LLM-Craft, a novel pipeline that leverages large language models (LLMs) to iteratively reason about and generate deformation-based crafting action sequences. We simplify and couple the state and action representations to further encourage shape-based reasoning. To the best of our knowledge, LLM-Craft is the first system successfully leveraging LLMs for complex deformable object interactions. Through our experiments, we demonstrate that with the LLM-Craft framework, LLMs are able to successfully create a set of simple letter shapes. We explore a variety of rollout strategies, and compare performances of LLM-Craft variants with and without an explicit goal shape images. For videos and prompting details, please visit our project website: https://sites.google.com/andrew.cmu.edu/llmcraft/home |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08648 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM-Craft: Robotic Crafting of Elasto-Plastic Objects with Large Language Models Bartsch, Alison Farimani, Amir Barati Robotics When humans create sculptures, we are able to reason about how geometrically we need to alter the clay state to reach our target goal. We are not computing point-wise similarity metrics, or reasoning about low-level positioning of our tools, but instead determining the higher-level changes that need to be made. In this work, we propose LLM-Craft, a novel pipeline that leverages large language models (LLMs) to iteratively reason about and generate deformation-based crafting action sequences. We simplify and couple the state and action representations to further encourage shape-based reasoning. To the best of our knowledge, LLM-Craft is the first system successfully leveraging LLMs for complex deformable object interactions. Through our experiments, we demonstrate that with the LLM-Craft framework, LLMs are able to successfully create a set of simple letter shapes. We explore a variety of rollout strategies, and compare performances of LLM-Craft variants with and without an explicit goal shape images. For videos and prompting details, please visit our project website: https://sites.google.com/andrew.cmu.edu/llmcraft/home |
| title | LLM-Craft: Robotic Crafting of Elasto-Plastic Objects with Large Language Models |
| topic | Robotics |
| url | https://arxiv.org/abs/2406.08648 |