Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.17679 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913884154101760 |
|---|---|
| author | Xu, Zhu Zhao, Zhiqiang Zhang, Zihan Liu, Yuchi Shen, Quanwei Liu, Fei Kuang, Yu He, Jian Liu, Conglin |
| author_facet | Xu, Zhu Zhao, Zhiqiang Zhang, Zihan Liu, Yuchi Shen, Quanwei Liu, Fei Kuang, Yu He, Jian Liu, Conglin |
| contents | Tokenization methods like Byte-Pair Encoding (BPE) enhance computational efficiency in large language models (LLMs) but often obscure internal character structures within tokens. This limitation hinders LLMs' ability to predict precise character positions, which is crucial in tasks like Chinese Spelling Correction (CSC) where identifying the positions of misspelled characters accelerates correction processes. We propose Token Internal Position Awareness (TIPA), a method that significantly improves models' ability to capture character positions within tokens by training them on reverse character prediction tasks using the tokenizer's vocabulary. Experiments demonstrate that TIPA enhances position prediction accuracy in LLMs, enabling more precise identification of target characters in original text. Furthermore, when applied to downstream tasks that do not require exact position prediction, TIPA still boosts performance in tasks needing character-level information, validating its versatility and effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17679 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning Xu, Zhu Zhao, Zhiqiang Zhang, Zihan Liu, Yuchi Shen, Quanwei Liu, Fei Kuang, Yu He, Jian Liu, Conglin Computation and Language Tokenization methods like Byte-Pair Encoding (BPE) enhance computational efficiency in large language models (LLMs) but often obscure internal character structures within tokens. This limitation hinders LLMs' ability to predict precise character positions, which is crucial in tasks like Chinese Spelling Correction (CSC) where identifying the positions of misspelled characters accelerates correction processes. We propose Token Internal Position Awareness (TIPA), a method that significantly improves models' ability to capture character positions within tokens by training them on reverse character prediction tasks using the tokenizer's vocabulary. Experiments demonstrate that TIPA enhances position prediction accuracy in LLMs, enabling more precise identification of target characters in original text. Furthermore, when applied to downstream tasks that do not require exact position prediction, TIPA still boosts performance in tasks needing character-level information, validating its versatility and effectiveness. |
| title | Enhancing Character-Level Understanding in LLMs through Token Internal Structure Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.17679 |