Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913002067853312 |
|---|---|
| author | Du, Ruofei Hersh, Benjamin Li, David Numan, Nels Qian, Xun Chen, Yanhe Zhou, Zhongyi Chen, Xingyue Ren, Jiahao Bettridge, Robert Timothy Chen, Xiang 'Anthony' Faruqi, Faraz Toh, Steve Kim, David |
| author_facet | Du, Ruofei Hersh, Benjamin Li, David Numan, Nels Qian, Xun Chen, Yanhe Zhou, Zhongyi Chen, Xingyue Ren, Jiahao Bettridge, Robert Timothy Chen, Xiang 'Anthony' Faruqi, Faraz Toh, Steve Kim, David |
| contents | While large language models (LLMs) have accelerated 2D software development through intent-driven "vibe coding", prototyping intelligent Extended Reality (XR) experiences remains a major challenge. The fundamental barrier is not just the steep learning curve for human creators, but that low-level sensor APIs and complex game engine hierarchies are ill-suited for LLM reasoning, routinely exceeding context windows and inducing syntax hallucinations. To bridge this gap, we contribute XR Blocks, an open-source, LLM-native WebXR framework. Unlike traditional engines, XR Blocks introduces a semantic "Reality Model" that aligns spatial computing primitives (users, physical environments, and agents) with natural language, providing a robust, concise vocabulary optimized for generative AI. Building upon this foundation, we present Vibe Coding XR, an end-to-end prototyping workflow that leverages LLMs to translate high-level prompts (e.g., "create a dandelion that reacts to my hand") directly into functional, physics-aware mixed-reality applications. To minimize the friction of on-device testing, the workflow introduces a seamless desktop "simulated reality" to headset deployment loop. Finally, we introduce VCXR60, a pilot dataset of 60 XR prompts paired with an automated evaluation pipeline. Our technical evaluation demonstrates high one-shot execution success, enabling practitioners to bypass lowlevel hurdles and rapidly move from "idea to reality". Code and live demos are available at https://github.com/google/xrblocks and http://xrblocks.github.io/gem. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_24591 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini Du, Ruofei Hersh, Benjamin Li, David Numan, Nels Qian, Xun Chen, Yanhe Zhou, Zhongyi Chen, Xingyue Ren, Jiahao Bettridge, Robert Timothy Chen, Xiang 'Anthony' Faruqi, Faraz Toh, Steve Kim, David Human-Computer Interaction While large language models (LLMs) have accelerated 2D software development through intent-driven "vibe coding", prototyping intelligent Extended Reality (XR) experiences remains a major challenge. The fundamental barrier is not just the steep learning curve for human creators, but that low-level sensor APIs and complex game engine hierarchies are ill-suited for LLM reasoning, routinely exceeding context windows and inducing syntax hallucinations. To bridge this gap, we contribute XR Blocks, an open-source, LLM-native WebXR framework. Unlike traditional engines, XR Blocks introduces a semantic "Reality Model" that aligns spatial computing primitives (users, physical environments, and agents) with natural language, providing a robust, concise vocabulary optimized for generative AI. Building upon this foundation, we present Vibe Coding XR, an end-to-end prototyping workflow that leverages LLMs to translate high-level prompts (e.g., "create a dandelion that reacts to my hand") directly into functional, physics-aware mixed-reality applications. To minimize the friction of on-device testing, the workflow introduces a seamless desktop "simulated reality" to headset deployment loop. Finally, we introduce VCXR60, a pilot dataset of 60 XR prompts paired with an automated evaluation pipeline. Our technical evaluation demonstrates high one-shot execution success, enabling practitioners to bypass lowlevel hurdles and rapidly move from "idea to reality". Code and live demos are available at https://github.com/google/xrblocks and http://xrblocks.github.io/gem. |
| title | Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini |
| topic | Human-Computer Interaction |
| url | https://arxiv.org/abs/2603.24591 |