Bridging the gap between natural user expression with complex automation programming in smart homes
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910574721368064 |
|---|---|
| author | Shi, Yingtian Liu, Xiaoyi Yu, Chun Yang, Tianao Gao, Cheng Liang, Chen Shi, Yuanchun |
| author_facet | Shi, Yingtian Liu, Xiaoyi Yu, Chun Yang, Tianao Gao, Cheng Liang, Chen Shi, Yuanchun |
| contents | A long-standing challenge in end-user programming (EUP) is to trade off between natural user expression and the complexity of programming tasks. As large language models (LLMs) are empowered to handle semantic inference and natural language understanding, it remains under-explored how such capabilities can facilitate end-users to configure complex automation more naturally and easily. We propose AwareAuto, an EUP system that standardizes user expression and finishes two-step inference with the LLMs to achieve automation generation. AwareAuto allows contextual, multi-modality, and flexible user expression to configure complex automation tasks (e.g., dynamic parameters, multiple conditional branches, and temporal constraints), which are non-manageable in traditional EUP solutions. By studying realistic, complex rules data, AwareAuto gains 91.7% accuracy in matching user intentions and feasibility. We introduced user interaction to ensure system controllability and usability. We discuss the opportunities and challenges of incorporating LLMs in end-user programming techniques and grounding complex smart home contexts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_12687 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Bridging the gap between natural user expression with complex automation programming in smart homes Shi, Yingtian Liu, Xiaoyi Yu, Chun Yang, Tianao Gao, Cheng Liang, Chen Shi, Yuanchun Human-Computer Interaction A long-standing challenge in end-user programming (EUP) is to trade off between natural user expression and the complexity of programming tasks. As large language models (LLMs) are empowered to handle semantic inference and natural language understanding, it remains under-explored how such capabilities can facilitate end-users to configure complex automation more naturally and easily. We propose AwareAuto, an EUP system that standardizes user expression and finishes two-step inference with the LLMs to achieve automation generation. AwareAuto allows contextual, multi-modality, and flexible user expression to configure complex automation tasks (e.g., dynamic parameters, multiple conditional branches, and temporal constraints), which are non-manageable in traditional EUP solutions. By studying realistic, complex rules data, AwareAuto gains 91.7% accuracy in matching user intentions and feasibility. We introduced user interaction to ensure system controllability and usability. We discuss the opportunities and challenges of incorporating LLMs in end-user programming techniques and grounding complex smart home contexts. |
| title | Bridging the gap between natural user expression with complex automation programming in smart homes |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2408.12687 |