LLM Vocabulary Compression for Low-Compute Environments
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909383137427456 |
|---|---|
| author | Vennam, Sreeram Joishy, Anish Kumaraguru, Ponnurangam |
| author_facet | Vennam, Sreeram Joishy, Anish Kumaraguru, Ponnurangam |
| contents | We present a method to compress the final linear layer of language models, reducing memory usage by up to 3.4x without significant performance loss. By grouping tokens based on Byte Pair Encoding (BPE) merges, we prevent materialization of the memory-intensive logits tensor. Evaluations on the TinyStories dataset show that our method performs on par with GPT-Neo and GPT2 while significantly improving throughput by up to 3x, making it suitable for low-compute environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_06371 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM Vocabulary Compression for Low-Compute Environments Vennam, Sreeram Joishy, Anish Kumaraguru, Ponnurangam Computation and Language Machine Learning I.2.6; I.2.7 We present a method to compress the final linear layer of language models, reducing memory usage by up to 3.4x without significant performance loss. By grouping tokens based on Byte Pair Encoding (BPE) merges, we prevent materialization of the memory-intensive logits tensor. Evaluations on the TinyStories dataset show that our method performs on par with GPT-Neo and GPT2 while significantly improving throughput by up to 3x, making it suitable for low-compute environments. |
| title | LLM Vocabulary Compression for Low-Compute Environments |
| topic | Computation and Language Machine Learning I.2.6; I.2.7 |
| url | https://arxiv.org/abs/2411.06371 |