Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.16037 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913041026646016 |
|---|---|
| author | Steger, Sophie Li, Rui Ennadir, Sofiane Sims, Anya Solin, Arno Pernkopf, Franz Trapp, Martin |
| author_facet | Steger, Sophie Li, Rui Ennadir, Sofiane Sims, Anya Solin, Arno Pernkopf, Franz Trapp, Martin |
| contents | The widespread adoption of large language models (LLMs) has increased concerns about their robustness. Vulnerabilities in perturbations of tokenisation of the input indicate that models trained with a deterministic canonical tokenisation can be brittle to adversarial attacks. Recent studies suggest that stochastic tokenisation can deliver internal representations that are less sensitive to perturbations. In this paper, we analyse how stochastic tokenisations affect robustness to adversarial attacks and random perturbations. We systematically study this over a range of learning regimes (pre-training, supervised fine-tuning, and in-context learning), data sets, and model architectures. We show that pre-training and fine-tuning with uniformly sampled stochastic tokenisations improve robustness to random and adversarial perturbations. Evaluating on uniformly sampled non-canonical tokenisations reduces the accuracy of a canonically trained Llama-1b model by 29.8%. We find that training with stochastic tokenisation preserves accuracy without increasing inference cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_16037 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Stochasticity in Tokenisation Improves Robustness Steger, Sophie Li, Rui Ennadir, Sofiane Sims, Anya Solin, Arno Pernkopf, Franz Trapp, Martin Computation and Language The widespread adoption of large language models (LLMs) has increased concerns about their robustness. Vulnerabilities in perturbations of tokenisation of the input indicate that models trained with a deterministic canonical tokenisation can be brittle to adversarial attacks. Recent studies suggest that stochastic tokenisation can deliver internal representations that are less sensitive to perturbations. In this paper, we analyse how stochastic tokenisations affect robustness to adversarial attacks and random perturbations. We systematically study this over a range of learning regimes (pre-training, supervised fine-tuning, and in-context learning), data sets, and model architectures. We show that pre-training and fine-tuning with uniformly sampled stochastic tokenisations improve robustness to random and adversarial perturbations. Evaluating on uniformly sampled non-canonical tokenisations reduces the accuracy of a canonically trained Llama-1b model by 29.8%. We find that training with stochastic tokenisation preserves accuracy without increasing inference cost. |
| title | Stochasticity in Tokenisation Improves Robustness |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.16037 |