Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2510.10560 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918159211036672 |
|---|---|
| author | Aman, Euhid Carlin, Esteban Pao, Hsing-Kuo Beltrame, Giovanni Sari, Ghaluh Indah Permata Chen, Yie-Tarng |
| author_facet | Aman, Euhid Carlin, Esteban Pao, Hsing-Kuo Beltrame, Giovanni Sari, Ghaluh Indah Permata Chen, Yie-Tarng |
| contents | Cross-attention transformers and other multimodal vision-language models excel at grounding and generation; however, their extensive, full-precision backbones make it challenging to deploy them on edge devices. Memory-augmented architectures enhance the utilization of past context; however, most works rarely pair them with aggressive edge-oriented quantization. We introduce BitMar, a quantized multimodal transformer that proposes an external human-like episodic memory for effective image-text generation on hardware with limited resources. BitMar utilizes 1.58-bit encoders, one for text (BitNet-style) and one for vision (DiNOv2-based), to create compact embeddings that are combined and used to query a fixed-size key-value episodic memory. During vector retrieval, the BitNet decoder applies per-layer conditioning, which increases the contextual relevance of generated content. The decoder also employs attention sinks with a sliding-window mechanism to process long or streaming inputs under tight memory budgets. The combination of per-layer conditioning and sliding-window attention achieves a strong quality-speed trade-off, delivering competitive captioning and multimodal understanding at low latency with a small model footprint. These characteristics make BitMar well-suited for edge deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10560 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices Aman, Euhid Carlin, Esteban Pao, Hsing-Kuo Beltrame, Giovanni Sari, Ghaluh Indah Permata Chen, Yie-Tarng Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition 68T50 I.2.7 Cross-attention transformers and other multimodal vision-language models excel at grounding and generation; however, their extensive, full-precision backbones make it challenging to deploy them on edge devices. Memory-augmented architectures enhance the utilization of past context; however, most works rarely pair them with aggressive edge-oriented quantization. We introduce BitMar, a quantized multimodal transformer that proposes an external human-like episodic memory for effective image-text generation on hardware with limited resources. BitMar utilizes 1.58-bit encoders, one for text (BitNet-style) and one for vision (DiNOv2-based), to create compact embeddings that are combined and used to query a fixed-size key-value episodic memory. During vector retrieval, the BitNet decoder applies per-layer conditioning, which increases the contextual relevance of generated content. The decoder also employs attention sinks with a sliding-window mechanism to process long or streaming inputs under tight memory budgets. The combination of per-layer conditioning and sliding-window attention achieves a strong quality-speed trade-off, delivering competitive captioning and multimodal understanding at low latency with a small model footprint. These characteristics make BitMar well-suited for edge deployment. |
| title | BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition 68T50 I.2.7 |
| url | https://arxiv.org/abs/2510.10560 |