WorldSimBench: Towards Video Generation Models as World Simulators
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910663502200832 |
|---|---|
| author | Qin, Yiran Shi, Zhelun Yu, Jiwen Wang, Xijun Zhou, Enshen Li, Lijun Yin, Zhenfei Liu, Xihui Sheng, Lu Shao, Jing Bai, Lei Ouyang, Wanli Zhang, Ruimao |
| author_facet | Qin, Yiran Shi, Zhelun Yu, Jiwen Wang, Xijun Zhou, Enshen Li, Lijun Yin, Zhenfei Liu, Xihui Sheng, Lu Shao, Jing Bai, Lei Ouyang, Wanli Zhang, Ruimao |
| contents | Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues to hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use to train a Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulators. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation-aware video can be accurately translated into the correct control signals in dynamic environments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_18072 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WorldSimBench: Towards Video Generation Models as World Simulators Qin, Yiran Shi, Zhelun Yu, Jiwen Wang, Xijun Zhou, Enshen Li, Lijun Yin, Zhenfei Liu, Xihui Sheng, Lu Shao, Jing Bai, Lei Ouyang, Wanli Zhang, Ruimao Computer Vision and Pattern Recognition Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues to hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use to train a Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulators. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation-aware video can be accurately translated into the correct control signals in dynamic environments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence. |
| title | WorldSimBench: Towards Video Generation Models as World Simulators |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.18072 |