Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916924939567104 |
|---|---|
| author | Yang, Han Lan, Jian Liu, Yihong Schütze, Hinrich Seidl, Thomas |
| author_facet | Yang, Han Lan, Jian Liu, Yihong Schütze, Hinrich Seidl, Thomas |
| contents | Autoregressive language models are vulnerable to orthographic attacks, where input text is perturbed with characters from multilingual alphabets, leading to substantial performance degradation. This vulnerability primarily stems from the out-of-vocabulary issue inherent in subword tokenizers and their embeddings. To address this limitation, we propose a pixel-based generative language model that replaces the text-based embeddings with pixel-based representations by rendering words as individual images. This design provides stronger robustness to noisy inputs, while an extension of compatibility to multilingual text across diverse writing systems. We evaluate the proposed method on the multilingual LAMBADA dataset, WMT24 dataset and the SST-2 benchmark, demonstrating both its resilience to orthographic noise and its effectiveness in multilingual settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_21206 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach Yang, Han Lan, Jian Liu, Yihong Schütze, Hinrich Seidl, Thomas Computation and Language Artificial Intelligence Autoregressive language models are vulnerable to orthographic attacks, where input text is perturbed with characters from multilingual alphabets, leading to substantial performance degradation. This vulnerability primarily stems from the out-of-vocabulary issue inherent in subword tokenizers and their embeddings. To address this limitation, we propose a pixel-based generative language model that replaces the text-based embeddings with pixel-based representations by rendering words as individual images. This design provides stronger robustness to noisy inputs, while an extension of compatibility to multilingual text across diverse writing systems. We evaluate the proposed method on the multilingual LAMBADA dataset, WMT24 dataset and the SST-2 benchmark, demonstrating both its resilience to orthographic noise and its effectiveness in multilingual settings. |
| title | Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2508.21206 |