MoST: Multi-modality Scene Tokenization for Motion Prediction
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911860195852288 |
|---|---|
| author | Mu, Norman Ji, Jingwei Yang, Zhenpei Harada, Nate Tang, Haotian Chen, Kan Qi, Charles R. Ge, Runzhou Goel, Kratarth Yang, Zoey Ettinger, Scott Al-Rfou, Rami Anguelov, Dragomir Zhou, Yin |
| author_facet | Mu, Norman Ji, Jingwei Yang, Zhenpei Harada, Nate Tang, Haotian Chen, Kan Qi, Charles R. Ge, Runzhou Goel, Kratarth Yang, Zoey Ettinger, Scott Al-Rfou, Rami Anguelov, Dragomir Zhou, Yin |
| contents | Many existing motion prediction approaches rely on symbolic perception outputs to generate agent trajectories, such as bounding boxes, road graph information and traffic lights. This symbolic representation is a high-level abstraction of the real world, which may render the motion prediction model vulnerable to perception errors (e.g., failures in detecting open-vocabulary obstacles) while missing salient information from the scene context (e.g., poor road conditions). An alternative paradigm is end-to-end learning from raw sensors. However, this approach suffers from the lack of interpretability and requires significantly more training resources. In this work, we propose tokenizing the visual world into a compact set of scene elements and then leveraging pre-trained image foundation models and LiDAR neural networks to encode all the scene elements in an open-vocabulary manner. The image foundation model enables our scene tokens to encode the general knowledge of the open world while the LiDAR neural network encodes geometry information. Our proposed representation can efficiently encode the multi-frame multi-modality observations with a few hundred tokens and is compatible with most transformer-based architectures. To evaluate our method, we have augmented Waymo Open Motion Dataset with camera embeddings. Experiments over Waymo Open Motion Dataset show that our approach leads to significant performance improvements over the state-of-the-art. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_19531 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MoST: Multi-modality Scene Tokenization for Motion Prediction Mu, Norman Ji, Jingwei Yang, Zhenpei Harada, Nate Tang, Haotian Chen, Kan Qi, Charles R. Ge, Runzhou Goel, Kratarth Yang, Zoey Ettinger, Scott Al-Rfou, Rami Anguelov, Dragomir Zhou, Yin Computer Vision and Pattern Recognition Many existing motion prediction approaches rely on symbolic perception outputs to generate agent trajectories, such as bounding boxes, road graph information and traffic lights. This symbolic representation is a high-level abstraction of the real world, which may render the motion prediction model vulnerable to perception errors (e.g., failures in detecting open-vocabulary obstacles) while missing salient information from the scene context (e.g., poor road conditions). An alternative paradigm is end-to-end learning from raw sensors. However, this approach suffers from the lack of interpretability and requires significantly more training resources. In this work, we propose tokenizing the visual world into a compact set of scene elements and then leveraging pre-trained image foundation models and LiDAR neural networks to encode all the scene elements in an open-vocabulary manner. The image foundation model enables our scene tokens to encode the general knowledge of the open world while the LiDAR neural network encodes geometry information. Our proposed representation can efficiently encode the multi-frame multi-modality observations with a few hundred tokens and is compatible with most transformer-based architectures. To evaluate our method, we have augmented Waymo Open Motion Dataset with camera embeddings. Experiments over Waymo Open Motion Dataset show that our approach leads to significant performance improvements over the state-of-the-art. |
| title | MoST: Multi-modality Scene Tokenization for Motion Prediction |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.19531 |