Glyph: Scaling Context Windows via Visual-Text Compression
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918165010710528 |
|---|---|
| author | Cheng, Jiale Liu, Yusen Zhang, Xinyu Fei, Yulin Hong, Wenyi Lyu, Ruiliang Wang, Weihan Su, Zhe Gu, Xiaotao Liu, Xiao Bai, Yushi Tang, Jie Wang, Hongning Huang, Minlie |
| author_facet | Cheng, Jiale Liu, Yusen Zhang, Xinyu Fei, Yulin Hong, Wenyi Lyu, Ruiliang Wang, Weihan Su, Zhe Gu, Xiaotao Liu, Xiao Bai, Yushi Tang, Jie Wang, Hongning Huang, Minlie |
| contents | Large language models (LLMs) increasingly rely on long-context modeling for tasks such as document understanding, code analysis, and multi-step reasoning. However, scaling context windows to the million-token level brings prohibitive computational and memory costs, limiting the practicality of long-context LLMs. In this work, we take a different perspective-visual context scaling-to tackle this challenge. Instead of extending token-based sequences, we propose Glyph, a framework that renders long texts into images and processes them with vision-language models (VLMs). This approach substantially compresses textual input while preserving semantic information, and we further design an LLM-driven genetic search to identify optimal visual rendering configurations for balancing accuracy and compression. Through extensive experiments, we demonstrate that our method achieves 3-4x token compression while maintaining accuracy comparable to leading LLMs such as Qwen3-8B on various long-context benchmarks. This compression also leads to around 4x faster prefilling and decoding, and approximately 2x faster SFT training. Furthermore, under extreme compression, a 128K-context VLM could scale to handle 1M-token-level text tasks. In addition, the rendered text data benefits real-world multimodal tasks, such as document understanding. Our code and model are released at https://github.com/thu-coai/Glyph. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_17800 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Glyph: Scaling Context Windows via Visual-Text Compression Cheng, Jiale Liu, Yusen Zhang, Xinyu Fei, Yulin Hong, Wenyi Lyu, Ruiliang Wang, Weihan Su, Zhe Gu, Xiaotao Liu, Xiao Bai, Yushi Tang, Jie Wang, Hongning Huang, Minlie Computer Vision and Pattern Recognition Computation and Language Machine Learning Large language models (LLMs) increasingly rely on long-context modeling for tasks such as document understanding, code analysis, and multi-step reasoning. However, scaling context windows to the million-token level brings prohibitive computational and memory costs, limiting the practicality of long-context LLMs. In this work, we take a different perspective-visual context scaling-to tackle this challenge. Instead of extending token-based sequences, we propose Glyph, a framework that renders long texts into images and processes them with vision-language models (VLMs). This approach substantially compresses textual input while preserving semantic information, and we further design an LLM-driven genetic search to identify optimal visual rendering configurations for balancing accuracy and compression. Through extensive experiments, we demonstrate that our method achieves 3-4x token compression while maintaining accuracy comparable to leading LLMs such as Qwen3-8B on various long-context benchmarks. This compression also leads to around 4x faster prefilling and decoding, and approximately 2x faster SFT training. Furthermore, under extreme compression, a 128K-context VLM could scale to handle 1M-token-level text tasks. In addition, the rendered text data benefits real-world multimodal tasks, such as document understanding. Our code and model are released at https://github.com/thu-coai/Glyph. |
| title | Glyph: Scaling Context Windows via Visual-Text Compression |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2510.17800 |