Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model System
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911099765391360 |
|---|---|
| author | Chen, Yiye Sawhney, Harpreet Gydé, Nicholas Jian, Yanan Saunders, Jack Vela, Patricio Lundell, Ben |
| author_facet | Chen, Yiye Sawhney, Harpreet Gydé, Nicholas Jian, Yanan Saunders, Jack Vela, Patricio Lundell, Ben |
| contents | Scene graphs have emerged as a structured and serializable environment representation for grounded spatial reasoning with Large Language Models (LLMs). In this work, we propose SG^2, an iterative Schema-Guided Scene-Graph reasoning framework based on multi-agent LLMs. The agents are grouped into two modules: a (1) Reasoner module for abstract task planning and graph information queries generation, and a (2) Retriever module for extracting corresponding graph information based on code-writing following the queries. Two modules collaborate iteratively, enabling sequential reasoning and adaptive attention to graph information. The scene graph schema, prompted to both modules, serves to not only streamline both reasoning and retrieval process, but also guide the cooperation between two modules. This eliminates the need to prompt LLMs with full graph data, reducing the chance of hallucination due to irrelevant information. Through experiments in multiple simulation environments, we show that our framework surpasses existing LLM-based approaches and baseline single-agent, tool-based Reason-while-Retrieve strategy in numerical Q\&A and planning tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03450 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model System Chen, Yiye Sawhney, Harpreet Gydé, Nicholas Jian, Yanan Saunders, Jack Vela, Patricio Lundell, Ben Machine Learning Artificial Intelligence Multiagent Systems Robotics Scene graphs have emerged as a structured and serializable environment representation for grounded spatial reasoning with Large Language Models (LLMs). In this work, we propose SG^2, an iterative Schema-Guided Scene-Graph reasoning framework based on multi-agent LLMs. The agents are grouped into two modules: a (1) Reasoner module for abstract task planning and graph information queries generation, and a (2) Retriever module for extracting corresponding graph information based on code-writing following the queries. Two modules collaborate iteratively, enabling sequential reasoning and adaptive attention to graph information. The scene graph schema, prompted to both modules, serves to not only streamline both reasoning and retrieval process, but also guide the cooperation between two modules. This eliminates the need to prompt LLMs with full graph data, reducing the chance of hallucination due to irrelevant information. Through experiments in multiple simulation environments, we show that our framework surpasses existing LLM-based approaches and baseline single-agent, tool-based Reason-while-Retrieve strategy in numerical Q\&A and planning tasks. |
| title | Schema-Guided Scene-Graph Reasoning based on Multi-Agent Large Language Model System |
| topic | Machine Learning Artificial Intelligence Multiagent Systems Robotics |
| url | https://arxiv.org/abs/2502.03450 |