Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914932219445248 |
|---|---|
| author | Chen, Yingfa Hu, Chenlong Feng, Cong Song, Chenyang Yu, Shi Han, Xu Liu, Zhiyuan Sun, Maosong |
| author_facet | Chen, Yingfa Hu, Chenlong Feng, Cong Song, Chenyang Yu, Shi Han, Xu Liu, Zhiyuan Sun, Maosong |
| contents | This study presents a multi-modal multi-granularity tokenizer specifically designed for analyzing ancient Chinese scripts, focusing on the Chu bamboo slip (CBS) script used during the Spring and Autumn and Warring States period (771-256 BCE) in Ancient China. Considering the complex hierarchical structure of ancient Chinese scripts, where a single character may be a combination of multiple sub-characters, our tokenizer first adopts character detection to locate character boundaries, and then conducts character recognition at both the character and sub-character levels. Moreover, to support the academic community, we have also assembled the first large-scale dataset of CBSs with over 100K annotated character image scans. On the part-of-speech tagging task built on our dataset, using our tokenizer gives a 5.5% relative improvement in F1-score compared to mainstream sub-word tokenizers. Our work not only aids in further investigations of the specific script but also has the potential to advance research on other forms of ancient Chinese scripts. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_01011 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts Chen, Yingfa Hu, Chenlong Feng, Cong Song, Chenyang Yu, Shi Han, Xu Liu, Zhiyuan Sun, Maosong Computation and Language Computer Vision and Pattern Recognition This study presents a multi-modal multi-granularity tokenizer specifically designed for analyzing ancient Chinese scripts, focusing on the Chu bamboo slip (CBS) script used during the Spring and Autumn and Warring States period (771-256 BCE) in Ancient China. Considering the complex hierarchical structure of ancient Chinese scripts, where a single character may be a combination of multiple sub-characters, our tokenizer first adopts character detection to locate character boundaries, and then conducts character recognition at both the character and sub-character levels. Moreover, to support the academic community, we have also assembled the first large-scale dataset of CBSs with over 100K annotated character image scans. On the part-of-speech tagging task built on our dataset, using our tokenizer gives a 5.5% relative improvement in F1-score compared to mainstream sub-word tokenizers. Our work not only aids in further investigations of the specific script but also has the potential to advance research on other forms of ancient Chinese scripts. |
| title | Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.01011 |