UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911505568497664 |
|---|---|
| author | Zhang, Yecheng Zhao, Rong Sha, Zhizhou Li, Yong Wang, Lei Hou, Ce Ji, Wen Huang, Hao Wan, Yunshan Yu, Jian Xia, Junhao Zhang, Yuru Shi, Chunlei |
| author_facet | Zhang, Yecheng Zhao, Rong Sha, Zhizhou Li, Yong Wang, Lei Hou, Ce Ji, Wen Huang, Hao Wan, Yunshan Yu, Jian Xia, Junhao Zhang, Yuru Shi, Chunlei |
| contents | Vision-language models (VLMs) can describe urban scenes in rich detail, yet consistently fail to produce reliable human preference labels in domain-specific tasks such as safety assessment and aesthetic evaluation. The standard fix, fine-tuning or RLHF, requires large-scale annotations and model retraining. We ask a different question: can a frozen VLM be aligned with human preferences without modifying any weights? Our key insight is that VLMs are strong concept extractors but poor decision calibrators. We propose a three-stage post-hoc pipeline that exploits this asymmetry: (i) interpretable evaluation dimensions are automatically mined from consensus exemplars; (ii) an Observer-Debater-Judge chain extracts robust concept scores from the frozen VLM; and (iii) locally-weighted ridge regression on a hybrid manifold calibrates these scores to human ratings. Applied as UrbanAlign on Place Pulse 2.0, the framework reaches 72.2% accuracy (kappa=0.45) across six perception categories, outperforming all baselines by +11.0 pp and zero-shot VLM by +15.5 pp, with full interpretability and zero weight modification. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_19442 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment Zhang, Yecheng Zhao, Rong Sha, Zhizhou Li, Yong Wang, Lei Hou, Ce Ji, Wen Huang, Hao Wan, Yunshan Yu, Jian Xia, Junhao Zhang, Yuru Shi, Chunlei Computer Vision and Pattern Recognition 68T45 I.2.10 Vision-language models (VLMs) can describe urban scenes in rich detail, yet consistently fail to produce reliable human preference labels in domain-specific tasks such as safety assessment and aesthetic evaluation. The standard fix, fine-tuning or RLHF, requires large-scale annotations and model retraining. We ask a different question: can a frozen VLM be aligned with human preferences without modifying any weights? Our key insight is that VLMs are strong concept extractors but poor decision calibrators. We propose a three-stage post-hoc pipeline that exploits this asymmetry: (i) interpretable evaluation dimensions are automatically mined from consensus exemplars; (ii) an Observer-Debater-Judge chain extracts robust concept scores from the frozen VLM; and (iii) locally-weighted ridge regression on a hybrid manifold calibrates these scores to human ratings. Applied as UrbanAlign on Place Pulse 2.0, the framework reaches 72.2% accuracy (kappa=0.45) across six perception categories, outperforming all baselines by +11.0 pp and zero-shot VLM by +15.5 pp, with full interpretability and zero weight modification. |
| title | UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment |
| topic | Computer Vision and Pattern Recognition 68T45 I.2.10 |
| url | https://arxiv.org/abs/2602.19442 |