Entropy-and-Channel-Aware Adaptive-Rate Semantic Communication with MLLM-Aided Feature Compensation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917328834265088 |
|---|---|
| author | Chen, Weixuan Yang, Qianqian Chen, Yuhao Huang, Chongwen Wang, Qian Xiong, Zehui Zhang, Zhaoyang |
| author_facet | Chen, Weixuan Yang, Qianqian Chen, Yuhao Huang, Chongwen Wang, Qian Xiong, Zehui Zhang, Zhaoyang |
| contents | Despite the transmission efficiency gains of semantic communication (SemCom) over traditional methods, most existing SemCom schemes still operate at a fixed transmission rate regardless of channel conditions and transmitted content, resulting in wasted resources in favorable channels and degraded performance in harsh channels. To address this issue, we propose a novel SemCom framework that incorporates an entropy-and-channel-aware adaptive rate control mechanism over MIMO Rayleigh fading channels. Specifically, we embed a joint representation of the channel state information (CSI) and the signal-to-noise ratio (SNR) into both the semantic encoder and decoder, thereby realizing channel-aware semantic coding and decoding. Moreover, the proposed method jointly exploits the CSI, the SNR, the feature maps, and their 2D entropy via two policy networks to selectively transmit only a subset of feature maps and, within each selected feature map, only a subset of symbols. Thereby, it achieves finer-grained adaptive rate control than existing methods. At the receiver, leveraging the strong visual understanding capability of multimodal large language models (MLLMs), we deploy the lightweight visual encoder (InternViT-300M) of the pre-trained InternVL3.5 model to compensate for discarded feature maps and symbols, and we fine-tune InternViT using low-rank adaptation (LoRA) for parameter-efficient training. Experimental results show that, with a carefully designed channel-aware loss function, our system automatically allocates more communication resources under poor channels to enhance task performance while reducing resource usage under favorable channels and maintaining high task performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_15414 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Entropy-and-Channel-Aware Adaptive-Rate Semantic Communication with MLLM-Aided Feature Compensation Chen, Weixuan Yang, Qianqian Chen, Yuhao Huang, Chongwen Wang, Qian Xiong, Zehui Zhang, Zhaoyang Image and Video Processing Despite the transmission efficiency gains of semantic communication (SemCom) over traditional methods, most existing SemCom schemes still operate at a fixed transmission rate regardless of channel conditions and transmitted content, resulting in wasted resources in favorable channels and degraded performance in harsh channels. To address this issue, we propose a novel SemCom framework that incorporates an entropy-and-channel-aware adaptive rate control mechanism over MIMO Rayleigh fading channels. Specifically, we embed a joint representation of the channel state information (CSI) and the signal-to-noise ratio (SNR) into both the semantic encoder and decoder, thereby realizing channel-aware semantic coding and decoding. Moreover, the proposed method jointly exploits the CSI, the SNR, the feature maps, and their 2D entropy via two policy networks to selectively transmit only a subset of feature maps and, within each selected feature map, only a subset of symbols. Thereby, it achieves finer-grained adaptive rate control than existing methods. At the receiver, leveraging the strong visual understanding capability of multimodal large language models (MLLMs), we deploy the lightweight visual encoder (InternViT-300M) of the pre-trained InternVL3.5 model to compensate for discarded feature maps and symbols, and we fine-tune InternViT using low-rank adaptation (LoRA) for parameter-efficient training. Experimental results show that, with a carefully designed channel-aware loss function, our system automatically allocates more communication resources under poor channels to enhance task performance while reducing resource usage under favorable channels and maintaining high task performance. |
| title | Entropy-and-Channel-Aware Adaptive-Rate Semantic Communication with MLLM-Aided Feature Compensation |
| topic | Image and Video Processing |
| url | https://arxiv.org/abs/2501.15414 |