Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.04125 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917123872260096 |
|---|---|
| author | Luo, Kerry Fu, Michael Peguero, Joshua Malik, Husnain Patil, Anvay Lin, Joyce Van Overborg, Megan Sarmiento, Ryan Zhu, Kevin |
| author_facet | Luo, Kerry Fu, Michael Peguero, Joshua Malik, Husnain Patil, Anvay Lin, Joyce Van Overborg, Megan Sarmiento, Ryan Zhu, Kevin |
| contents | Large language models (LLMs) have demonstrated several emergent behaviors with scale, including reasoning and fluency in long-form text generation. However, they continue to struggle with tasks requiring precise spatial and positional reasoning. ASCII art, a symbolic medium where characters encode structure and form, provides a unique probe of this limitation. We introduce ASCIIBench, a novel benchmark for evaluating both the generation and classification of ASCII-text images. ASCIIBench consists of a filtered dataset of 5,315 class-labeled ASCII images and is, to our knowledge, the first publicly available benchmark of its kind. Alongside the dataset, we release weights for a fine-tuned CLIP model adapted to capture ASCII structure, enabling the evaluation of LLM-generated ASCII art. Our analysis shows that cosine similarity over CLIP embeddings fails to separate most ASCII categories, yielding chance-level performance even for low-variance classes. In contrast, classes with high internal mean similarity exhibit clear discriminability, revealing that the bottleneck lies in representation rather than generational variance. These findings position ASCII art as a stress test for multimodal representations and motivate the development of new embedding methods or evaluation metrics tailored to symbolic visual modalities. All resources are available at https://github.com/ASCIIBench/ASCIIBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_04125 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text Luo, Kerry Fu, Michael Peguero, Joshua Malik, Husnain Patil, Anvay Lin, Joyce Van Overborg, Megan Sarmiento, Ryan Zhu, Kevin Machine Learning Large language models (LLMs) have demonstrated several emergent behaviors with scale, including reasoning and fluency in long-form text generation. However, they continue to struggle with tasks requiring precise spatial and positional reasoning. ASCII art, a symbolic medium where characters encode structure and form, provides a unique probe of this limitation. We introduce ASCIIBench, a novel benchmark for evaluating both the generation and classification of ASCII-text images. ASCIIBench consists of a filtered dataset of 5,315 class-labeled ASCII images and is, to our knowledge, the first publicly available benchmark of its kind. Alongside the dataset, we release weights for a fine-tuned CLIP model adapted to capture ASCII structure, enabling the evaluation of LLM-generated ASCII art. Our analysis shows that cosine similarity over CLIP embeddings fails to separate most ASCII categories, yielding chance-level performance even for low-variance classes. In contrast, classes with high internal mean similarity exhibit clear discriminability, revealing that the bottleneck lies in representation rather than generational variance. These findings position ASCII art as a stress test for multimodal representations and motivate the development of new embedding methods or evaluation metrics tailored to symbolic visual modalities. All resources are available at https://github.com/ASCIIBench/ASCIIBench. |
| title | ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2512.04125 |