Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908919370088448 |
|---|---|
| author | Zhang, Baoheng Liu, Jiahui Zhao, Gui Zhang, Weizhou Ma, Yixuan Jiang, Jun Chen, Yingxian Fok, Wilton W. T. Qi, Xiaojuan So, Hayden Kwok-Hay |
| author_facet | Zhang, Baoheng Liu, Jiahui Zhao, Gui Zhang, Weizhou Ma, Yixuan Jiang, Jun Chen, Yingxian Fok, Wilton W. T. Qi, Xiaojuan So, Hayden Kwok-Hay |
| contents | Multimodal Large Language Models (MLLMs) perform strong vision-language reasoning under standard conditions but fail in extreme illumination, where RGB inputs lose irrevocable structure and semantics. We propose Event-MLLM, an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames. Two key components drive our approach: an Illumination Indicator - a learnable signal derived from a DINOv2 branch that represents exposure degradation and adaptively modulates event-RGB fusion - and an Illumination Correction Loss that aligns fused features with non-degraded (normal-light) semantics in the latent space, compensating for information lost in extreme lighting. We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples (around 6 QA pairs each) across diverse scenes and 17 brightness rates (0.05x - 20x), plus an instruct-following benchmark for reasoning, counting, and fine-grained recognition under extreme lighting. Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines, setting a new state of the art in robust multimodal perception and reasoning under challenging illumination. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_27558 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models Zhang, Baoheng Liu, Jiahui Zhao, Gui Zhang, Weizhou Ma, Yixuan Jiang, Jun Chen, Yingxian Fok, Wilton W. T. Qi, Xiaojuan So, Hayden Kwok-Hay Computer Vision and Pattern Recognition Multimodal Large Language Models (MLLMs) perform strong vision-language reasoning under standard conditions but fail in extreme illumination, where RGB inputs lose irrevocable structure and semantics. We propose Event-MLLM, an event-enhanced model that performs all-light visual reasoning by dynamically fusing event streams with RGB frames. Two key components drive our approach: an Illumination Indicator - a learnable signal derived from a DINOv2 branch that represents exposure degradation and adaptively modulates event-RGB fusion - and an Illumination Correction Loss that aligns fused features with non-degraded (normal-light) semantics in the latent space, compensating for information lost in extreme lighting. We curate the first multi-illumination event-instruction corpus for MLLMs, with 2,241 event-RGB samples (around 6 QA pairs each) across diverse scenes and 17 brightness rates (0.05x - 20x), plus an instruct-following benchmark for reasoning, counting, and fine-grained recognition under extreme lighting. Experiments show that Event-MLLM markedly outperforms general-purpose, illumination-adaptive, and event-only baselines, setting a new state of the art in robust multimodal perception and reasoning under challenging illumination. |
| title | Learning to See through Illumination Extremes with Event Streaming in Multimodal Large Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.27558 |