Token Alignment via Character Matching for Subword Completion
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913264135307264 |
|---|---|
| author | Athiwaratkun, Ben Wang, Shiqi Shang, Mingyue Tian, Yuchen Wang, Zijian Gonugondla, Sujan Kumar Gouda, Sanjay Krishna Kwiatowski, Rob Nallapati, Ramesh Xiang, Bing |
| author_facet | Athiwaratkun, Ben Wang, Shiqi Shang, Mingyue Tian, Yuchen Wang, Zijian Gonugondla, Sujan Kumar Gouda, Sanjay Krishna Kwiatowski, Rob Nallapati, Ramesh Xiang, Bing |
| contents | Generative models, widely utilized in various applications, can often struggle with prompts corresponding to partial tokens. This struggle stems from tokenization, where partial tokens fall out of distribution during inference, leading to incorrect or nonsensical outputs. This paper examines a technique to alleviate the tokenization artifact on text completion in generative models, maintaining performance even in regular non-subword cases. The method, termed token alignment, involves backtracking to the last complete tokens and ensuring the model's generation aligns with the prompt. This approach showcases marked improvement across many partial token scenarios, including nuanced cases like space-prefix and partial indentation, with only a minor time increase. The technique and analysis detailed in this paper contribute to the continuous advancement of generative models in handling partial inputs, bearing relevance for applications like code completion and text autocompletion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_08688 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Token Alignment via Character Matching for Subword Completion Athiwaratkun, Ben Wang, Shiqi Shang, Mingyue Tian, Yuchen Wang, Zijian Gonugondla, Sujan Kumar Gouda, Sanjay Krishna Kwiatowski, Rob Nallapati, Ramesh Xiang, Bing Computation and Language Artificial Intelligence Generative models, widely utilized in various applications, can often struggle with prompts corresponding to partial tokens. This struggle stems from tokenization, where partial tokens fall out of distribution during inference, leading to incorrect or nonsensical outputs. This paper examines a technique to alleviate the tokenization artifact on text completion in generative models, maintaining performance even in regular non-subword cases. The method, termed token alignment, involves backtracking to the last complete tokens and ensuring the model's generation aligns with the prompt. This approach showcases marked improvement across many partial token scenarios, including nuanced cases like space-prefix and partial indentation, with only a minor time increase. The technique and analysis detailed in this paper contribute to the continuous advancement of generative models in handling partial inputs, bearing relevance for applications like code completion and text autocompletion. |
| title | Token Alignment via Character Matching for Subword Completion |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2403.08688 |