Image de couverture de livre

Back to Bytes: Revisiting Tokenization Through UTF-8

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Moryossef, Amit, Meister, Clara, Stepachev, Pavel, Elliott, Desmond
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

UTF-8 Plumbing: Byte-level Tokenizers Unavoidably Enable LLMs to Generate Ill-formed UTF-8
par: Firestone, Preston, et autres
Publié: (2025)

sign.mt: Real-Time Multilingual Sign Language Translation Application
par: Moryossef, Amit
Publié: (2023)

Real-Time Multilingual Sign Language Processing
par: Moryossef, Amit
Publié: (2024)

SignBank+: Preparing a Multilingual Sign Language Dataset for Machine Translation Using Large Language Models
par: Moryossef, Amit, et autres
Publié: (2023)

signwriting-evaluation: Effective Sign Language Evaluation via SignWriting
par: Moryossef, Amit, et autres
Publié: (2024)

Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
par: Foroutan, Negar, et autres
Publié: (2025)

Token Distillation: Attention-aware Input Embeddings For New Tokens
par: Dobler, Konstantin, et autres
Publié: (2025)

Context and System Fusion in Post-ASR Emotion Recognition with Large Language Models
par: Stepachev, Pavel, et autres
Publié: (2024)

Tracking Universal Features Through Fine-Tuning and Model Merging
par: Horn, Niels, et autres
Publié: (2024)

Pose-Based Sign Language Appearance Transfer
par: Moryossef, Amit, et autres
Publié: (2024)

Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era
par: Oneata, Dan, et autres
Publié: (2025)

What Language is This? Ask Your Tokenizer
par: Meister, Clara, et autres
Publié: (2026)

Ham2Pose: Animating Sign Language Notation into Pose Sequences
par: Shalev-Arkushin, Rotem, et autres
Publié: (2022)

A Formal Perspective on Byte-Pair Encoding
par: Zouhar, Vilém, et autres
Publié: (2023)

Optimizing Hand Region Detection in MediaPipe Holistic Full-Body Pose Estimation to Improve Accuracy and Avoid Downstream Errors
par: Moryossef, Amit
Publié: (2024)

How to Compute the Probability of a Word
par: Pimentel, Tiago, et autres
Publié: (2024)

Quality or Quantity? On Data Scale and Diversity in Adapting Large Language Models for Low-Resource Translation
par: Iyer, Vivek, et autres
Publié: (2024)

Byte BPE Tokenization as an Inverse string Homomorphism
par: Geng, Saibo, et autres
Publié: (2024)

Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizers
par: Jang, Eugene, et autres
Publié: (2024)

SignCLIP: Connecting Text and Sign Language by Contrastive Learning
par: Jiang, Zifan, et autres
Publié: (2024)

CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation
par: Ziegler, Ingo, et autres
Publié: (2024)

Entropy-Driven Pre-Tokenization for Byte-Pair Encoding
par: Hu, Yifan, et autres
Publié: (2025)

Distilling Token-Trained Models into Byte-Level Models
par: Bao, Zishuo, et autres
Publié: (2026)

ByteFlow: Language Modeling through Adaptive Byte Compression without a Tokenizer
par: Deng, Chunyuan, et autres
Publié: (2026)

Towards a Similarity-adjusted Surprisal Theory
par: Meister, Clara, et autres
Publié: (2024)

Byte Latent Transformer: Patches Scale Better Than Tokens
par: Pagnoni, Artidoro, et autres
Publié: (2024)

CroCo: Cross-Lingual Contrastive Preference Tuning on Self-Generations
par: Zhang, Mike, et autres
Publié: (2026)

Cross-Tokenizer LLM Distillation through a Byte-Level Interface
par: Singh, Avyav Kumar, et autres
Publié: (2026)

How Do Multilingual Language Models Remember Facts?
par: Fierro, Constanza, et autres
Publié: (2024)

Validating UTF-8 In Less Than One Instruction Per Byte
par: Keiser, John, et autres
Publié: (2020)

From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution
par: Chizhov, Pavel, et autres
Publié: (2026)

Kathleen: Oscillator-Based Byte-Level Text Classification Without Tokenization or Attention
par: Fountzoulas, George
Publié: (2026)

MambaByte: Token-free Selective State Space Model
par: Wang, Junxiong, et autres
Publié: (2024)

BBPE16: UTF-16-based byte-level byte-pair encoding for improved multilingual speech recognition
par: Kim, Hyunsik, et autres
Publié: (2026)

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
par: Gigant, Théo, et autres
Publié: (2026)

ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Models
par: Villegas, Danae Sánchez, et autres
Publié: (2025)

Can Community Notes Replace Professional Fact-Checkers?
par: Borenstein, Nadav, et autres
Publié: (2025)

GPUTOK: GPU Accelerated Byte Level BPE Tokenization
par: Kadamba, Venu Gopal, et autres
Publié: (2026)

MultimodalHugs: Enabling Sign Language Processing in Hugging Face
par: Sant, Gerard, et autres
Publié: (2025)

From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
par: Zhang, Wanpeng, et autres
Publié: (2024)