The Invisible Mentor: Inferring User Actions from Screen Recordings to Recommend Better Workflows
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911613544562688 |
|---|---|
| author | Yan, Litao Head, Andrew Milne, Ken Le, Vu Gulwani, Sumit Parnin, Chris Murphy-Hill, Emerson |
| author_facet | Yan, Litao Head, Andrew Milne, Ken Le, Vu Gulwani, Sumit Parnin, Chris Murphy-Hill, Emerson |
| contents | Many users struggle to notice when a more efficient workflow exists in feature-rich tools like Excel. Existing AI assistants offer help only after users describe their goals or problems, which can be effortful and imprecise. We present InvisibleMentor, a system that turns screen recordings of task completion into vision-grounded reflections on tasks. It detects issues such as repetitive edits and recommends more efficient alternatives based on observed behavior. Unlike prior systems that rely on logs, APIs, or user prompts, InvisibleMentor operates directly on screen recordings. It uses a two-stage pipeline: a vision-language model reconstructs actions and context, and a language model generates structured, high-fidelity suggestions. In evaluation, InvisibleMentor accurately identified inefficient workflows, and participants found its suggestions more actionable, tailored, and more helpful for learning and improvement compared to a prompt-based spreadsheet assistant. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_26557 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Invisible Mentor: Inferring User Actions from Screen Recordings to Recommend Better Workflows Yan, Litao Head, Andrew Milne, Ken Le, Vu Gulwani, Sumit Parnin, Chris Murphy-Hill, Emerson Human-Computer Interaction H.5.2 Many users struggle to notice when a more efficient workflow exists in feature-rich tools like Excel. Existing AI assistants offer help only after users describe their goals or problems, which can be effortful and imprecise. We present InvisibleMentor, a system that turns screen recordings of task completion into vision-grounded reflections on tasks. It detects issues such as repetitive edits and recommends more efficient alternatives based on observed behavior. Unlike prior systems that rely on logs, APIs, or user prompts, InvisibleMentor operates directly on screen recordings. It uses a two-stage pipeline: a vision-language model reconstructs actions and context, and a language model generates structured, high-fidelity suggestions. In evaluation, InvisibleMentor accurately identified inefficient workflows, and participants found its suggestions more actionable, tailored, and more helpful for learning and improvement compared to a prompt-based spreadsheet assistant. |
| title | The Invisible Mentor: Inferring User Actions from Screen Recordings to Recommend Better Workflows |
| topic | Human-Computer Interaction H.5.2 |
| url | https://arxiv.org/abs/2509.26557 |