Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.09715 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909598437343232 |
|---|---|
| author | Ren, Mengmeng Qiao, Li Yang, Long Gao, Zhen Chen, Jian Mashhadi, Mahdi Boloursaz Xiao, Pei Tafazolli, Rahim Bennis, Mehdi |
| author_facet | Ren, Mengmeng Qiao, Li Yang, Long Gao, Zhen Chen, Jian Mashhadi, Mahdi Boloursaz Xiao, Pei Tafazolli, Rahim Bennis, Mehdi |
| contents | This paper develops an edge-device collaborative Generative Semantic Communications (Gen SemCom) framework leveraging pre-trained Multi-modal/Vision Language Models (M/VLMs) for ultra-low-rate semantic communication via textual prompts. The proposed framework optimizes the use of M/VLMs on the wireless edge/device to generate high-fidelity textual prompts through visual captioning/question answering, which are then transmitted over a wireless channel for SemCom. Specifically, we develop a multi-user Gen SemCom framework using pre-trained M/VLMs, and formulate a joint optimization problem of prompt generation offloading, communication and computation resource allocation to minimize the latency and maximize the resulting semantic quality. Due to the nonconvex nature of the problem with highly coupled discrete and continuous variables, we decompose it as a two-level problem and propose a low-complexity swap/leaving/joining (SLJ)-based matching algorithm. Simulation results demonstrate significant performance improvements over the conventional semanticunaware/non-collaborative offloading benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_09715 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generative Semantic Communication via Textual Prompts: Latency Performance Tradeoffs Ren, Mengmeng Qiao, Li Yang, Long Gao, Zhen Chen, Jian Mashhadi, Mahdi Boloursaz Xiao, Pei Tafazolli, Rahim Bennis, Mehdi Information Theory Computer Science and Game Theory This paper develops an edge-device collaborative Generative Semantic Communications (Gen SemCom) framework leveraging pre-trained Multi-modal/Vision Language Models (M/VLMs) for ultra-low-rate semantic communication via textual prompts. The proposed framework optimizes the use of M/VLMs on the wireless edge/device to generate high-fidelity textual prompts through visual captioning/question answering, which are then transmitted over a wireless channel for SemCom. Specifically, we develop a multi-user Gen SemCom framework using pre-trained M/VLMs, and formulate a joint optimization problem of prompt generation offloading, communication and computation resource allocation to minimize the latency and maximize the resulting semantic quality. Due to the nonconvex nature of the problem with highly coupled discrete and continuous variables, we decompose it as a two-level problem and propose a low-complexity swap/leaving/joining (SLJ)-based matching algorithm. Simulation results demonstrate significant performance improvements over the conventional semanticunaware/non-collaborative offloading benchmarks. |
| title | Generative Semantic Communication via Textual Prompts: Latency Performance Tradeoffs |
| topic | Information Theory Computer Science and Game Theory |
| url | https://arxiv.org/abs/2409.09715 |