LMM-PCQA: Assisting Point Cloud Quality Assessment with LMM
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911978877878272 |
|---|---|
| author | Zhang, Zicheng Wu, Haoning Zhou, Yingjie Li, Chunyi Sun, Wei Chen, Chaofeng Min, Xiongkuo Liu, Xiaohong Lin, Weisi Zhai, Guangtao |
| author_facet | Zhang, Zicheng Wu, Haoning Zhou, Yingjie Li, Chunyi Sun, Wei Chen, Chaofeng Min, Xiongkuo Liu, Xiaohong Lin, Weisi Zhai, Guangtao |
| contents | Although large multi-modality models (LMMs) have seen extensive exploration and application in various quality assessment studies, their integration into Point Cloud Quality Assessment (PCQA) remains unexplored. Given LMMs' exceptional performance and robustness in low-level vision and quality assessment tasks, this study aims to investigate the feasibility of imparting PCQA knowledge to LMMs through text supervision. To achieve this, we transform quality labels into textual descriptions during the fine-tuning phase, enabling LMMs to derive quality rating logits from 2D projections of point clouds. To compensate for the loss of perception in the 3D domain, structural features are extracted as well. These quality logits and structural features are then combined and regressed into quality scores. Our experimental results affirm the effectiveness of our approach, showcasing a novel integration of LMMs into PCQA that enhances model understanding and assessment accuracy. We hope our contributions can inspire subsequent investigations into the fusion of LMMs with PCQA, fostering advancements in 3D visual quality analysis and beyond. The code is available at https://github.com/zzc-1998/LMM-PCQA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_18203 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LMM-PCQA: Assisting Point Cloud Quality Assessment with LMM Zhang, Zicheng Wu, Haoning Zhou, Yingjie Li, Chunyi Sun, Wei Chen, Chaofeng Min, Xiongkuo Liu, Xiaohong Lin, Weisi Zhai, Guangtao Computer Vision and Pattern Recognition Artificial Intelligence Although large multi-modality models (LMMs) have seen extensive exploration and application in various quality assessment studies, their integration into Point Cloud Quality Assessment (PCQA) remains unexplored. Given LMMs' exceptional performance and robustness in low-level vision and quality assessment tasks, this study aims to investigate the feasibility of imparting PCQA knowledge to LMMs through text supervision. To achieve this, we transform quality labels into textual descriptions during the fine-tuning phase, enabling LMMs to derive quality rating logits from 2D projections of point clouds. To compensate for the loss of perception in the 3D domain, structural features are extracted as well. These quality logits and structural features are then combined and regressed into quality scores. Our experimental results affirm the effectiveness of our approach, showcasing a novel integration of LMMs into PCQA that enhances model understanding and assessment accuracy. We hope our contributions can inspire subsequent investigations into the fusion of LMMs with PCQA, fostering advancements in 3D visual quality analysis and beyond. The code is available at https://github.com/zzc-1998/LMM-PCQA. |
| title | LMM-PCQA: Assisting Point Cloud Quality Assessment with LMM |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2404.18203 |