From Language Models over Tokens to Language Models over Characters
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909644678496256 |
|---|---|
| author | Vieira, Tim LeBrun, Ben Giulianelli, Mario Gastaldi, Juan Luis DuSell, Brian Terilla, John O'Donnell, Timothy J. Cotterell, Ryan |
| author_facet | Vieira, Tim LeBrun, Ben Giulianelli, Mario Gastaldi, Juan Luis DuSell, Brian Terilla, John O'Donnell, Timothy J. Cotterell, Ryan |
| contents | Modern language models are internally -- and mathematically -- distributions over $\it{token}$ strings rather than $\it{character}$ strings, posing numerous challenges for programmers building user applications on top of them. For example, if a prompt is specified as a character string, it must be tokenized before passing it to the token-level language model. Thus, the tokenizer and consequent processing are very sensitive to the specification of the prompt (e.g., whether the prompt ends with a space or not). This paper presents algorithms for converting token-level language models to character-level ones. We present both exact and approximate algorithms. In the empirical portion of the paper, we benchmark the practical runtime and approximation quality. Across four publicly available language models, we find that -- even with a small computation budget -- our method is able to accurately approximate the character-level distribution at reasonably fast speeds, and that a significant improvement in the language model's compression rate (bits/byte) is achieved. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_03719 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Language Models over Tokens to Language Models over Characters Vieira, Tim LeBrun, Ben Giulianelli, Mario Gastaldi, Juan Luis DuSell, Brian Terilla, John O'Donnell, Timothy J. Cotterell, Ryan Computation and Language Artificial Intelligence Modern language models are internally -- and mathematically -- distributions over $\it{token}$ strings rather than $\it{character}$ strings, posing numerous challenges for programmers building user applications on top of them. For example, if a prompt is specified as a character string, it must be tokenized before passing it to the token-level language model. Thus, the tokenizer and consequent processing are very sensitive to the specification of the prompt (e.g., whether the prompt ends with a space or not). This paper presents algorithms for converting token-level language models to character-level ones. We present both exact and approximate algorithms. In the empirical portion of the paper, we benchmark the practical runtime and approximation quality. Across four publicly available language models, we find that -- even with a small computation budget -- our method is able to accurately approximate the character-level distribution at reasonably fast speeds, and that a significant improvement in the language model's compression rate (bits/byte) is achieved. |
| title | From Language Models over Tokens to Language Models over Characters |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.03719 |