NeuralOS: Towards Simulating Operating Systems via Neural Generative Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918383927164928 |
|---|---|
| author | Rivard, Luke Sun, Sun Guo, Hongyu Chen, Wenhu Deng, Yuntian |
| author_facet | Rivard, Luke Sun, Sun Guo, Hongyu Chen, Wenhu Deng, Yuntian |
| contents | We introduce NeuralOS, a neural framework that simulates graphical user interfaces (GUIs) of operating systems by directly predicting screen frames in response to user inputs such as mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and realistic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably predicts state transitions like application launches. Beyond reproducing existing systems, NeuralOS shows that synthesized training data can teach the model to simulate applications that were never installed, as illustrated by a Doom application, and suggests a path toward learning user interfaces purely from synthetic demonstrations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_08800 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | NeuralOS: Towards Simulating Operating Systems via Neural Generative Models Rivard, Luke Sun, Sun Guo, Hongyu Chen, Wenhu Deng, Yuntian Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Human-Computer Interaction Machine Learning We introduce NeuralOS, a neural framework that simulates graphical user interfaces (GUIs) of operating systems by directly predicting screen frames in response to user inputs such as mouse movements, clicks, and keyboard events. NeuralOS combines a recurrent neural network (RNN), which tracks computer state, with a diffusion-based neural renderer that generates screen images. The model is trained on a dataset of Ubuntu XFCE recordings, which include both randomly generated interactions and realistic interactions produced by AI agents. Experiments show that NeuralOS successfully renders realistic GUI sequences, accurately captures mouse interactions, and reliably predicts state transitions like application launches. Beyond reproducing existing systems, NeuralOS shows that synthesized training data can teach the model to simulate applications that were never installed, as illustrated by a Doom application, and suggests a path toward learning user interfaces purely from synthetic demonstrations. |
| title | NeuralOS: Towards Simulating Operating Systems via Neural Generative Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2507.08800 |