Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.15707 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910024950874112 |
|---|---|
| author | Mahfuz, Rehana Guo, Yinyi Visser, Erik Chinchili, Phanidhar |
| author_facet | Mahfuz, Rehana Guo, Yinyi Visser, Erik Chinchili, Phanidhar |
| contents | Real-time conversational assistants for procedural tasks often depend on video input, which can be computationally expensive and compromise user privacy. For the first time, we propose a real-time conversational assistant that provides comprehensive guidance for a procedural task using only lightweight privacy-preserving modalities such as audio and IMU inputs from a user's wearable device to understand the context. This assistant proactively communicates step-by-step instructions to a user performing a furniture assembly task, and answers user questions. We construct a dataset containing conversations where the assistant guides the user in performing the task. On observing that an off-the-shelf language model is a very talkative assistant, we design a novel User Whim Agnostic (UWA) LoRA finetuning method which improves the model's ability to suppress less informative dialogues, while maintaining its tendency to communicate important instructions. This leads to >30% improvement in the F-score. Finetuning the model also results in a 16x speedup by eliminating the need to provide in-context examples in the prompt. We further describe how such an assistant is implemented on edge devices with no dependence on the cloud. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_15707 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU Mahfuz, Rehana Guo, Yinyi Visser, Erik Chinchili, Phanidhar Multimedia Computation and Language Machine Learning Real-time conversational assistants for procedural tasks often depend on video input, which can be computationally expensive and compromise user privacy. For the first time, we propose a real-time conversational assistant that provides comprehensive guidance for a procedural task using only lightweight privacy-preserving modalities such as audio and IMU inputs from a user's wearable device to understand the context. This assistant proactively communicates step-by-step instructions to a user performing a furniture assembly task, and answers user questions. We construct a dataset containing conversations where the assistant guides the user in performing the task. On observing that an off-the-shelf language model is a very talkative assistant, we design a novel User Whim Agnostic (UWA) LoRA finetuning method which improves the model's ability to suppress less informative dialogues, while maintaining its tendency to communicate important instructions. This leads to >30% improvement in the F-score. Finetuning the model also results in a 16x speedup by eliminating the need to provide in-context examples in the prompt. We further describe how such an assistant is implemented on edge devices with no dependence on the cloud. |
| title | Proactive Conversational Assistant for a Procedural Manual Task based on Audio and IMU |
| topic | Multimedia Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2602.15707 |