ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912671371100160 |
|---|---|
| author | Yang, Bufang Xu, Lilin Zeng, Liekang Liu, Kaiwei Jiang, Siyang Lu, Wenrui Chen, Hongkai Jiang, Xiaofan Xing, Guoliang Yan, Zhenyu |
| author_facet | Yang, Bufang Xu, Lilin Zeng, Liekang Liu, Kaiwei Jiang, Siyang Lu, Wenrui Chen, Hongkai Jiang, Xiaofan Xing, Guoliang Yan, Zhenyu |
| contents | Recent advances in Large Language Models (LLMs) have propelled intelligent agents from reactive responses to proactive support. While promising, existing proactive agents either rely exclusively on observations from enclosed environments (e.g., desktop UIs) with direct LLM inference or employ rule-based proactive notifications, leading to suboptimal user intent understanding and limited functionality for proactive service. In this paper, we introduce ContextAgent, the first context-aware proactive agent that incorporates extensive sensory contexts surrounding humans to enhance the proactivity of LLM agents. ContextAgent first extracts multi-dimensional contexts from massive sensory perceptions on wearables (e.g., video and audio) to understand user intentions. ContextAgent then leverages the sensory contexts and personas from historical data to predict the necessity for proactive services. When proactive assistance is needed, ContextAgent further automatically calls the necessary tools to assist users unobtrusively. To evaluate this new task, we curate ContextAgentBench, the first benchmark for evaluating context-aware proactive LLM agents, covering 1,000 samples across nine daily scenarios and twenty tools. Experiments on ContextAgentBench show that ContextAgent outperforms baselines by achieving up to 8.5% and 6.0% higher accuracy in proactive predictions and tool calling, respectively. We hope our research can inspire the development of more advanced, human-centric, proactive AI assistants. The code and dataset are publicly available at https://github.com/openaiotlab/ContextAgent. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14668 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions Yang, Bufang Xu, Lilin Zeng, Liekang Liu, Kaiwei Jiang, Siyang Lu, Wenrui Chen, Hongkai Jiang, Xiaofan Xing, Guoliang Yan, Zhenyu Artificial Intelligence Computation and Language Human-Computer Interaction Recent advances in Large Language Models (LLMs) have propelled intelligent agents from reactive responses to proactive support. While promising, existing proactive agents either rely exclusively on observations from enclosed environments (e.g., desktop UIs) with direct LLM inference or employ rule-based proactive notifications, leading to suboptimal user intent understanding and limited functionality for proactive service. In this paper, we introduce ContextAgent, the first context-aware proactive agent that incorporates extensive sensory contexts surrounding humans to enhance the proactivity of LLM agents. ContextAgent first extracts multi-dimensional contexts from massive sensory perceptions on wearables (e.g., video and audio) to understand user intentions. ContextAgent then leverages the sensory contexts and personas from historical data to predict the necessity for proactive services. When proactive assistance is needed, ContextAgent further automatically calls the necessary tools to assist users unobtrusively. To evaluate this new task, we curate ContextAgentBench, the first benchmark for evaluating context-aware proactive LLM agents, covering 1,000 samples across nine daily scenarios and twenty tools. Experiments on ContextAgentBench show that ContextAgent outperforms baselines by achieving up to 8.5% and 6.0% higher accuracy in proactive predictions and tool calling, respectively. We hope our research can inspire the development of more advanced, human-centric, proactive AI assistants. The code and dataset are publicly available at https://github.com/openaiotlab/ContextAgent. |
| title | ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions |
| topic | Artificial Intelligence Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2505.14668 |