Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrations
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915928521834496 |
|---|---|
| author | Kim, Edward He, Daniel Chao, Jorge Rajca, Wiktor Amin, Mohammed Malpani, Nishant Desai, Ruta Oulasvirta, Antti Hartmann, Bjoern Seshia, Sanjit |
| author_facet | Kim, Edward He, Daniel Chao, Jorge Rajca, Wiktor Amin, Mohammed Malpani, Nishant Desai, Ruta Oulasvirta, Antti Hartmann, Bjoern Seshia, Sanjit |
| contents | Teaching systems physical tasks is a long standing goal in HCI, yet most prior work has focused on non collaborative physical activities. Collaborative tasks introduce added complexity, requiring systems to infer users assumptions about their teammates intent, which is an inherently ambiguous and dynamic process. This necessitates representations that are interpretable and correctable, enabling users to inspect and refine system behavior. We address this challenge by framing collaborative task learning as a program synthesis problem. Our system represents behavior as editable programs and uses narrated demonstrations, i.e. paired physical actions and natural language, as a unified modality for teaching, inspecting, and correcting system logic without requiring users to see or write code. The same modality is used for the system to communicate its learning to users. In a within subjects study, 20 users taught multiplayer soccer tactics to our system. 70 percent (14/20) of participants successfully refined learned programs to match their intent and 90 percent (18/20) found it easy to correct the programs. The study surfaced unique challenges in representing learning as programs and in enabling users to teach collaborative physical activities. We discuss these issues and outline mitigation strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24250 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrations Kim, Edward He, Daniel Chao, Jorge Rajca, Wiktor Amin, Mohammed Malpani, Nishant Desai, Ruta Oulasvirta, Antti Hartmann, Bjoern Seshia, Sanjit Artificial Intelligence Human-Computer Interaction Machine Learning Teaching systems physical tasks is a long standing goal in HCI, yet most prior work has focused on non collaborative physical activities. Collaborative tasks introduce added complexity, requiring systems to infer users assumptions about their teammates intent, which is an inherently ambiguous and dynamic process. This necessitates representations that are interpretable and correctable, enabling users to inspect and refine system behavior. We address this challenge by framing collaborative task learning as a program synthesis problem. Our system represents behavior as editable programs and uses narrated demonstrations, i.e. paired physical actions and natural language, as a unified modality for teaching, inspecting, and correcting system logic without requiring users to see or write code. The same modality is used for the system to communicate its learning to users. In a within subjects study, 20 users taught multiplayer soccer tactics to our system. 70 percent (14/20) of participants successfully refined learned programs to match their intent and 90 percent (18/20) found it easy to correct the programs. The study surfaced unique challenges in representing learning as programs and in enabling users to teach collaborative physical activities. We discuss these issues and outline mitigation strategies. |
| title | Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrations |
| topic | Artificial Intelligence Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2509.24250 |