Cover Image

OmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Jiahao Nick, Xu, Yan, Grossman, Tovi, Santosa, Stephanie, Li, Michelle
Format:	Preprint
Published:	2024
Subjects:	Human-Computer Interaction Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

OmniQuery: Contextually Augmenting Captured Multimodal Memory to Enable Personal Question Answering
by: Li, Jiahao Nick, et al.
Published: (2024)

LOOM: Personalized Learning Informed by Daily LLM Conversations Toward Long-Term Mastery via a Dynamic Learner Memory Graph
by: Cui, Justin, et al.
Published: (2025)

CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs
by: Kazemitabaar, Majeed, et al.
Published: (2024)

StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagation
by: Suh, Sangho, et al.
Published: (2025)

Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learning
by: Kazemitabaar, Majeed, et al.
Published: (2024)

KnowledgeTrail: Generative Timeline for Exploration and Sensemaking of Historical Events and Knowledge Formation
by: Suh, Sangho, et al.
Published: (2025)

Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition
by: Kazemitabaar, Majeed, et al.
Published: (2024)

SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and Exploration
by: Brade, Stephen, et al.
Published: (2023)

Assistance or Disruption? Exploring and Evaluating the Design and Trade-offs of Proactive AI Programming Support
by: Pu, Kevin, et al.
Published: (2025)

Desk2Desk: Optimization-based Mixed Reality Workspace Integration for Remote Side-by-side Collaboration
by: Sidenmark, Ludwig, et al.
Published: (2024)

Towards Real-World Validity in Generative AI Benchmarks: Understanding and Designing Domain-Centered Evaluations for Journalism Practitioners
by: Li, Charlotte, et al.
Published: (2025)

Characterizing Human Actions in the Digital Platform by Temporal Context
by: Matsui, Akira, et al.
Published: (2022)

OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions
by: Luo, Cheng, et al.
Published: (2025)

Script2Screen: Supporting Dialogue Scriptwriting with Interactive Audiovisual Generation
by: Wang, Zhecheng, et al.
Published: (2025)

ACE, Action and Control via Explanations: A Proposal for LLMs to Provide Human-Centered Explainability for Multimodal AI Assistants
by: Watkins, Elizabeth Anne, et al.
Published: (2025)

MaRginalia: Enabling In-person Lecture Capturing and Note-taking Through Mixed Reality
by: Qiu, Leping, et al.
Published: (2025)

Adaptique: Multi-objective and Context-aware Online Adaptation of Selection Techniques in Virtual Reality
by: Lai, Chao-Jung, et al.
Published: (2025)

ImageInThat: Manipulating Images to Convey User Instructions to Robots
by: Mahadevan, Karthik, et al.
Published: (2025)

When LLMs Can't Help: Real-World Evaluation of LLMs in Nutrition
by: Li, Karen Jia-Hui, et al.
Published: (2025)

IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedback
by: Pu, Kevin, et al.
Published: (2024)

TRAFA: Anticipating User Actions to Reduce Errors in Procedural Tasks with Predictive Feedback
by: Mokhtar, Sassan, et al.
Published: (2026)

You Only Look at Screens: Multimodal Chain-of-Action Agents
by: Zhang, Zhuosheng, et al.
Published: (2023)

Mind the Value-Action Gap: Do LLMs Act in Alignment with Their Values?
by: Shen, Hua, et al.
Published: (2025)

An HCAI Methodological Framework (HCAI-MF): Putting It Into Action to Enable Human-Centered AI
by: Xu, Wei, et al.
Published: (2023)

ContextAgent: Context-Aware Proactive LLM Agents with Open-World Sensory Perceptions
by: Yang, Bufang, et al.
Published: (2025)

Addressing Climate Action Misperceptions with Generative AI
by: Remshard, Miriam, et al.
Published: (2026)

OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
by: Henry, Felix, et al.
Published: (2026)

Personagram: Bridging Personas and Product Design for Creative Ideation with Multimodal LLMs
by: Kim, Taewook, et al.
Published: (2026)

Think-Then-React: Towards Unconstrained Human Action-to-Reaction Generation
by: Tan, Wenhui, et al.
Published: (2025)

See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMs
by: Li, Zhimin, et al.
Published: (2025)

OpenOmni: A Collaborative Open Source Tool for Building Future-Ready Multimodal Conversational Agents
by: Sun, Qiang, et al.
Published: (2024)

See2Refine: Vision-Language Feedback Improves LLM-Based eHMI Action Designers
by: Xia, Ding, et al.
Published: (2026)

MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces
by: Luera, Reuben A., et al.
Published: (2025)

ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models
by: Reza, Mohi, et al.
Published: (2023)

Simulating Human Behavior with the Psychological-mechanism Agent: Integrating Feeling, Thought, and Action
by: Dong, Qing, et al.
Published: (2025)

Context-Value-Action Architecture for Value-Driven Large Language Model Agents
by: Zhang, TianZe, et al.
Published: (2026)

MeetMap: Real-Time Collaborative Dialogue Mapping with LLMs in Online Meetings
by: Chen, Xinyue, et al.
Published: (2025)

Harms in Repurposing Real-World Sensory Cues for Mixed Reality: A Causal Perspective
by: Tao, Yujie, et al.
Published: (2024)

Developing Situational Awareness for Joint Action with Autonomous Vehicles
by: Kaufman, Robert, et al.
Published: (2024)

Automating eHMI Action Design with LLMs for Automated Vehicle Communication
by: Xia, Ding, et al.
Published: (2025)