Embodied Understanding of Driving Scenarios
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914707483394048 |
|---|---|
| author | Zhou, Yunsong Huang, Linyan Bu, Qingwen Zeng, Jia Li, Tianyu Qiu, Hang Zhu, Hongzi Guo, Minyi Qiao, Yu Li, Hongyang |
| author_facet | Zhou, Yunsong Huang, Linyan Bu, Qingwen Zeng, Jia Li, Tianyu Qiu, Hang Zhu, Hongzi Guo, Minyi Qiao, Yu Li, Hongyang |
| contents | Embodied scene understanding serves as the cornerstone for autonomous agents to perceive, interpret, and respond to open driving scenarios. Such understanding is typically founded upon Vision-Language Models (VLMs). Nevertheless, existing VLMs are restricted to the 2D domain, devoid of spatial awareness and long-horizon extrapolation proficiencies. We revisit the key aspects of autonomous driving and formulate appropriate rubrics. Hereby, we introduce the Embodied Language Model (ELM), a comprehensive framework tailored for agents' understanding of driving scenes with large spatial and temporal spans. ELM incorporates space-aware pre-training to endow the agent with robust spatial localization capabilities. Besides, the model employs time-aware token selection to accurately inquire about temporal cues. We instantiate ELM on the reformulated multi-faced benchmark, and it surpasses previous state-of-the-art approaches in all aspects. All code, data, and models will be publicly shared. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_04593 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Embodied Understanding of Driving Scenarios Zhou, Yunsong Huang, Linyan Bu, Qingwen Zeng, Jia Li, Tianyu Qiu, Hang Zhu, Hongzi Guo, Minyi Qiao, Yu Li, Hongyang Computer Vision and Pattern Recognition Embodied scene understanding serves as the cornerstone for autonomous agents to perceive, interpret, and respond to open driving scenarios. Such understanding is typically founded upon Vision-Language Models (VLMs). Nevertheless, existing VLMs are restricted to the 2D domain, devoid of spatial awareness and long-horizon extrapolation proficiencies. We revisit the key aspects of autonomous driving and formulate appropriate rubrics. Hereby, we introduce the Embodied Language Model (ELM), a comprehensive framework tailored for agents' understanding of driving scenes with large spatial and temporal spans. ELM incorporates space-aware pre-training to endow the agent with robust spatial localization capabilities. Besides, the model employs time-aware token selection to accurately inquire about temporal cues. We instantiate ELM on the reformulated multi-faced benchmark, and it surpasses previous state-of-the-art approaches in all aspects. All code, data, and models will be publicly shared. |
| title | Embodied Understanding of Driving Scenarios |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.04593 |