Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.17185 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913403420803072 |
|---|---|
| author | Akabe, Koichi Kanda, Shunsuke Oda, Yusuke Mori, Shinsuke |
| author_facet | Akabe, Koichi Kanda, Shunsuke Oda, Yusuke Mori, Shinsuke |
| contents | This paper proposes an approach to improve the runtime efficiency of Japanese tokenization based on the pointwise linear classification (PLC) framework, which formulates the whole tokenization process as a sequence of linear classification problems. Our approach optimizes tokenization by leveraging the characteristics of the PLC framework and the task definition. Our approach involves (1) composing multiple classifications into array-based operations, (2) efficient feature lookup with memory-optimized automata, and (3) three orthogonal pre-processing methods for reducing actual score calculation. Thus, our approach makes the tokenization speed 5.7 times faster than the current approach based on the same model without decreasing tokenization accuracy. Our implementation is available at https://github.com/daac-tools/vaporetto under the MIT or Apache-2.0 license. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_17185 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Vaporetto: Efficient Japanese Tokenization Based on Improved Pointwise Linear Classification Akabe, Koichi Kanda, Shunsuke Oda, Yusuke Mori, Shinsuke Computation and Language This paper proposes an approach to improve the runtime efficiency of Japanese tokenization based on the pointwise linear classification (PLC) framework, which formulates the whole tokenization process as a sequence of linear classification problems. Our approach optimizes tokenization by leveraging the characteristics of the PLC framework and the task definition. Our approach involves (1) composing multiple classifications into array-based operations, (2) efficient feature lookup with memory-optimized automata, and (3) three orthogonal pre-processing methods for reducing actual score calculation. Thus, our approach makes the tokenization speed 5.7 times faster than the current approach based on the same model without decreasing tokenization accuracy. Our implementation is available at https://github.com/daac-tools/vaporetto under the MIT or Apache-2.0 license. |
| title | Vaporetto: Efficient Japanese Tokenization Based on Improved Pointwise Linear Classification |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.17185 |