Multi-word Tokenization for Sequence Compression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916194358919168 |
|---|---|
| author | Gee, Leonidas Rigutini, Leonardo Ernandes, Marco Zugarini, Andrea |
| author_facet | Gee, Leonidas Rigutini, Leonardo Ernandes, Marco Zugarini, Andrea |
| contents | Large Language Models have proven highly successful at modelling a variety of tasks. However, this comes at a steep computational cost that hinders wider industrial uptake. In this paper, we present MWT: a Multi-Word Tokenizer that goes beyond word boundaries by representing frequent multi-word expressions as single tokens. MWTs produce a more compact and efficient tokenization that yields two benefits: (1) Increase in performance due to a greater coverage of input data given a fixed sequence length budget; (2) Faster and lighter inference due to the ability to reduce the sequence length with negligible drops in performance. Our results show that MWT is more robust across shorter sequence lengths, thus allowing for major speedups via early sequence truncation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_09949 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multi-word Tokenization for Sequence Compression Gee, Leonidas Rigutini, Leonardo Ernandes, Marco Zugarini, Andrea Computation and Language Machine Learning Large Language Models have proven highly successful at modelling a variety of tasks. However, this comes at a steep computational cost that hinders wider industrial uptake. In this paper, we present MWT: a Multi-Word Tokenizer that goes beyond word boundaries by representing frequent multi-word expressions as single tokens. MWTs produce a more compact and efficient tokenization that yields two benefits: (1) Increase in performance due to a greater coverage of input data given a fixed sequence length budget; (2) Faster and lighter inference due to the ability to reduce the sequence length with negligible drops in performance. Our results show that MWT is more robust across shorter sequence lengths, thus allowing for major speedups via early sequence truncation. |
| title | Multi-word Tokenization for Sequence Compression |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2402.09949 |