CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenario
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916236554665984 |
|---|---|
| author | Duan, Zhizhao Cheng, Hao Xu, Duo Wu, Xi Zhang, Xiangxie Ye, Xi Xie, Zhen |
| author_facet | Duan, Zhizhao Cheng, Hao Xu, Duo Wu, Xi Zhang, Xiangxie Ye, Xi Xie, Zhen |
| contents | In the vast and dynamic landscape of urban settings, Traffic Safety Description and Analysis plays a pivotal role in applications ranging from insurance inspection to accident prevention. This paper introduces CityLLaVA, a novel fine-tuning framework for Visual Language Models (VLMs) designed for urban scenarios. CityLLaVA enhances model comprehension and prediction accuracy through (1) employing bounding boxes for optimal visual data preprocessing, including video best-view selection and visual prompt engineering during both training and testing phases; (2) constructing concise Question-Answer sequences and designing textual prompts to refine instruction comprehension; (3) implementing block expansion to fine-tune large VLMs efficiently; and (4) advancing prediction accuracy via a unique sequential questioning-based prediction augmentation. Demonstrating top-tier performance, our method achieved a benchmark score of 33.4308, securing the leading position on the leaderboard. The code can be found: https://github.com/alibaba/AICITY2024_Track2_AliOpenTrek_CityLLaVA |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_03194 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenario Duan, Zhizhao Cheng, Hao Xu, Duo Wu, Xi Zhang, Xiangxie Ye, Xi Xie, Zhen Computer Vision and Pattern Recognition In the vast and dynamic landscape of urban settings, Traffic Safety Description and Analysis plays a pivotal role in applications ranging from insurance inspection to accident prevention. This paper introduces CityLLaVA, a novel fine-tuning framework for Visual Language Models (VLMs) designed for urban scenarios. CityLLaVA enhances model comprehension and prediction accuracy through (1) employing bounding boxes for optimal visual data preprocessing, including video best-view selection and visual prompt engineering during both training and testing phases; (2) constructing concise Question-Answer sequences and designing textual prompts to refine instruction comprehension; (3) implementing block expansion to fine-tune large VLMs efficiently; and (4) advancing prediction accuracy via a unique sequential questioning-based prediction augmentation. Demonstrating top-tier performance, our method achieved a benchmark score of 33.4308, securing the leading position on the leaderboard. The code can be found: https://github.com/alibaba/AICITY2024_Track2_AliOpenTrek_CityLLaVA |
| title | CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenario |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2405.03194 |