Byte Latent Transformer: Patches Scale Better Than Tokens
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913610637246464 |
|---|---|
| author | Pagnoni, Artidoro Pasunuru, Ram Rodriguez, Pedro Nguyen, John Muller, Benjamin Li, Margaret Zhou, Chunting Yu, Lili Weston, Jason Zettlemoyer, Luke Ghosh, Gargi Lewis, Mike Holtzman, Ari Iyer, Srinivasan |
| author_facet | Pagnoni, Artidoro Pasunuru, Ram Rodriguez, Pedro Nguyen, John Muller, Benjamin Li, Margaret Zhou, Chunting Yu, Lili Weston, Jason Zettlemoyer, Luke Ghosh, Gargi Lewis, Mike Holtzman, Ari Iyer, Srinivasan |
| contents | We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_09871 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Byte Latent Transformer: Patches Scale Better Than Tokens Pagnoni, Artidoro Pasunuru, Ram Rodriguez, Pedro Nguyen, John Muller, Benjamin Li, Margaret Zhou, Chunting Yu, Lili Weston, Jason Zettlemoyer, Luke Ghosh, Gargi Lewis, Mike Holtzman, Ari Iyer, Srinivasan Computation and Language We introduce the Byte Latent Transformer (BLT), a new byte-level LLM architecture that, for the first time, matches tokenization-based LLM performance at scale with significant improvements in inference efficiency and robustness. BLT encodes bytes into dynamically sized patches, which serve as the primary units of computation. Patches are segmented based on the entropy of the next byte, allocating more compute and model capacity where increased data complexity demands it. We present the first FLOP controlled scaling study of byte-level models up to 8B parameters and 4T training bytes. Our results demonstrate the feasibility of scaling models trained on raw bytes without a fixed vocabulary. Both training and inference efficiency improve due to dynamically selecting long patches when data is predictable, along with qualitative improvements on reasoning and long tail generalization. Overall, for fixed inference costs, BLT shows significantly better scaling than tokenization-based models, by simultaneously growing both patch and model size. |
| title | Byte Latent Transformer: Patches Scale Better Than Tokens |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.09871 |