From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909532157902848 |
|---|---|
| author | Zhang, Wanpeng Xie, Zilong Feng, Yicheng Li, Yijiang Xing, Xingrun Zheng, Sipeng Lu, Zongqing |
| author_facet | Zhang, Wanpeng Xie, Zilong Feng, Yicheng Li, Yijiang Xing, Xingrun Zheng, Sipeng Lu, Zongqing |
| contents | Multimodal Large Language Models have made significant strides in integrating visual and textual information, yet they often struggle with effectively aligning these modalities. We introduce a novel image tokenizer that bridges this gap by applying the principle of Byte-Pair Encoding (BPE) to visual data. Unlike conventional approaches that rely on separate visual encoders, our method directly incorporates structural prior information into image tokens, mirroring the successful tokenization strategies used in text-only Large Language Models. This innovative approach enables Transformer models to more effectively learn and reason across modalities. Through theoretical analysis and extensive experiments, we demonstrate that our BPE Image Tokenizer significantly enhances MLLMs' multimodal understanding capabilities, even with limited training data. Leveraging this method, we develop Being-VL-0, a model that demonstrates superior performance across various benchmarks and shows promising scalability, potentially paving the way for more efficient and capable multimodal foundation models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02155 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities Zhang, Wanpeng Xie, Zilong Feng, Yicheng Li, Yijiang Xing, Xingrun Zheng, Sipeng Lu, Zongqing Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Multimodal Large Language Models have made significant strides in integrating visual and textual information, yet they often struggle with effectively aligning these modalities. We introduce a novel image tokenizer that bridges this gap by applying the principle of Byte-Pair Encoding (BPE) to visual data. Unlike conventional approaches that rely on separate visual encoders, our method directly incorporates structural prior information into image tokens, mirroring the successful tokenization strategies used in text-only Large Language Models. This innovative approach enables Transformer models to more effectively learn and reason across modalities. Through theoretical analysis and extensive experiments, we demonstrate that our BPE Image Tokenizer significantly enhances MLLMs' multimodal understanding capabilities, even with limited training data. Leveraging this method, we develop Being-VL-0, a model that demonstrates superior performance across various benchmarks and shows promising scalability, potentially paving the way for more efficient and capable multimodal foundation models. |
| title | From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.02155 |