PerspAct: Enhancing LLM Situated Collaboration Skills through Perspective Taking and Active Vision
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909898081566720 |
|---|---|
| author | Patania, Sabrina Annese, Luca Pellegrini, Anita Serino, Silvia Lambiase, Anna Pallonetto, Luca Rossi, Silvia Colombani, Simone Foulsham, Tom Ruggeri, Azzurra Ognibene, Dimitri |
| author_facet | Patania, Sabrina Annese, Luca Pellegrini, Anita Serino, Silvia Lambiase, Anna Pallonetto, Luca Rossi, Silvia Colombani, Simone Foulsham, Tom Ruggeri, Azzurra Ognibene, Dimitri |
| contents | Recent advances in Large Language Models (LLMs) and multimodal foundation models have significantly broadened their application in robotics and collaborative systems. However, effective multi-agent interaction necessitates robust perspective-taking capabilities, enabling models to interpret both physical and epistemic viewpoints. Current training paradigms often neglect these interactive contexts, resulting in challenges when models must reason about the subjectivity of individual perspectives or navigate environments with multiple observers. This study evaluates whether explicitly incorporating diverse points of view using the ReAct framework, an approach that integrates reasoning and acting, can enhance an LLM's ability to understand and ground the demands of other agents. We extend the classic Director task by introducing active visual exploration across a suite of seven scenarios of increasing perspective-taking complexity. These scenarios are designed to challenge the agent's capacity to resolve referential ambiguity based on visual access and interaction, under varying state representations and prompting strategies, including ReAct-style reasoning. Our results demonstrate that explicit perspective cues, combined with active exploration strategies, significantly improve the model's interpretative accuracy and collaborative effectiveness. These findings highlight the potential of integrating active perception with perspective-taking mechanisms in advancing LLMs' application in robotics and multi-agent systems, setting a foundation for future research into adaptive and context-aware AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_08098 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PerspAct: Enhancing LLM Situated Collaboration Skills through Perspective Taking and Active Vision Patania, Sabrina Annese, Luca Pellegrini, Anita Serino, Silvia Lambiase, Anna Pallonetto, Luca Rossi, Silvia Colombani, Simone Foulsham, Tom Ruggeri, Azzurra Ognibene, Dimitri Robotics Artificial Intelligence Computation and Language Human-Computer Interaction I.2.9; I.2.10; I.2.7; J.4 Recent advances in Large Language Models (LLMs) and multimodal foundation models have significantly broadened their application in robotics and collaborative systems. However, effective multi-agent interaction necessitates robust perspective-taking capabilities, enabling models to interpret both physical and epistemic viewpoints. Current training paradigms often neglect these interactive contexts, resulting in challenges when models must reason about the subjectivity of individual perspectives or navigate environments with multiple observers. This study evaluates whether explicitly incorporating diverse points of view using the ReAct framework, an approach that integrates reasoning and acting, can enhance an LLM's ability to understand and ground the demands of other agents. We extend the classic Director task by introducing active visual exploration across a suite of seven scenarios of increasing perspective-taking complexity. These scenarios are designed to challenge the agent's capacity to resolve referential ambiguity based on visual access and interaction, under varying state representations and prompting strategies, including ReAct-style reasoning. Our results demonstrate that explicit perspective cues, combined with active exploration strategies, significantly improve the model's interpretative accuracy and collaborative effectiveness. These findings highlight the potential of integrating active perception with perspective-taking mechanisms in advancing LLMs' application in robotics and multi-agent systems, setting a foundation for future research into adaptive and context-aware AI systems. |
| title | PerspAct: Enhancing LLM Situated Collaboration Skills through Perspective Taking and Active Vision |
| topic | Robotics Artificial Intelligence Computation and Language Human-Computer Interaction I.2.9; I.2.10; I.2.7; J.4 |
| url | https://arxiv.org/abs/2511.08098 |