Matrix3D: Large Photogrammetry Model All-in-One
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917964346818560 |
|---|---|
| author | Lu, Yuanxun Zhang, Jingyang Fang, Tian Nahmias, Jean-Daniel Tsin, Yanghai Quan, Long Cao, Xun Yao, Yao Li, Shiwei |
| author_facet | Lu, Yuanxun Zhang, Jingyang Fang, Tian Nahmias, Jean-Daniel Tsin, Yanghai Quan, Long Cao, Xun Yao, Yao Li, Shiwei |
| contents | We present Matrix3D, a unified model that performs several photogrammetry subtasks, including pose estimation, depth prediction, and novel view synthesis using just the same model. Matrix3D utilizes a multi-modal diffusion transformer (DiT) to integrate transformations across several modalities, such as images, camera parameters, and depth maps. The key to Matrix3D's large-scale multi-modal training lies in the incorporation of a mask learning strategy. This enables full-modality model training even with partially complete data, such as bi-modality data of image-pose and image-depth pairs, thus significantly increases the pool of available training data. Matrix3D demonstrates state-of-the-art performance in pose estimation and novel view synthesis tasks. Additionally, it offers fine-grained control through multi-round interactions, making it an innovative tool for 3D content creation. Project page: https://nju-3dv.github.io/projects/matrix3d. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_07685 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Matrix3D: Large Photogrammetry Model All-in-One Lu, Yuanxun Zhang, Jingyang Fang, Tian Nahmias, Jean-Daniel Tsin, Yanghai Quan, Long Cao, Xun Yao, Yao Li, Shiwei Computer Vision and Pattern Recognition We present Matrix3D, a unified model that performs several photogrammetry subtasks, including pose estimation, depth prediction, and novel view synthesis using just the same model. Matrix3D utilizes a multi-modal diffusion transformer (DiT) to integrate transformations across several modalities, such as images, camera parameters, and depth maps. The key to Matrix3D's large-scale multi-modal training lies in the incorporation of a mask learning strategy. This enables full-modality model training even with partially complete data, such as bi-modality data of image-pose and image-depth pairs, thus significantly increases the pool of available training data. Matrix3D demonstrates state-of-the-art performance in pose estimation and novel view synthesis tasks. Additionally, it offers fine-grained control through multi-round interactions, making it an innovative tool for 3D content creation. Project page: https://nju-3dv.github.io/projects/matrix3d. |
| title | Matrix3D: Large Photogrammetry Model All-in-One |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.07685 |