Image de couverture de livre

BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Chizhov, Pavel, Arnett, Catherine, Korotkova, Elizaveta, Yamshchikov, Ivan P.
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

From Where Words Come: Efficient Regularization of Code Tokenizers Through Source Attribution
par: Chizhov, Pavel, et autres
Publié: (2026)

Teaching Old Tokenizers New Words: Efficient Tokenizer Adaptation for Pre-trained Models
par: Purason, Taido, et autres
Publié: (2025)

Surface Fairness, Deep Bias: A Comparative Study of Bias in Language Models
par: Sorokovikova, Aleksandra, et autres
Publié: (2025)

BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization
par: Land, Sander, et autres
Publié: (2025)

Toxicity of the Commons: Curating Open-Source Pre-Training Data
par: Arnett, Catherine, et autres
Publié: (2024)

What the HellaSwag? On the Validity of Common-Sense Reasoning Benchmarks
par: Chizhov, Pavel, et autres
Publié: (2025)

Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training
par: Langlais, Pierre-Carl, et autres
Publié: (2025)

Vocabulary Transfer for Biomedical Texts: Add Tokens if You Can Not Add Data
par: Singh, Priyanka, et autres
Publié: (2022)

The Company You Keep: How LLMs Respond to Dark Triad Traits
par: Lu, Zeyi, et autres
Publié: (2026)

BlockBPE: Parallel BPE Tokenization
par: You, Amos
Publié: (2025)

Beyond Toxic: Toxicity Detection Datasets are Not Enough for Brand Safety
par: Korotkova, Elizaveta, et autres
Publié: (2023)

Adaptive BPE Tokenization for Enhanced Vocabulary Adaptation in Finetuning Pretrained Language Models
par: Balde, Gunjan, et autres
Publié: (2024)

Model in Distress: Sentiment Analysis on French Synthetic Social Media
par: Langlais, Pierre-Carl, et autres
Publié: (2026)

Batching BPE Tokenization Merges
par: Morgan, Alexander P.
Publié: (2024)

An Analysis of BPE Vocabulary Trimming in Neural Machine Translation
par: Cognetta, Marco, et autres
Publié: (2024)

Constructing a BPE Tokenization DFA
par: Berglund, Martin, et autres
Publié: (2024)

Byte BPE Tokenization as an Inverse string Homomorphism
par: Geng, Saibo, et autres
Publié: (2024)

CleanComedy: Creating Friendly Humor through Generative Techniques
par: Vikhorev, Dmitry, et autres
Publié: (2024)

LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers
par: Sun, Yike, et autres
Publié: (2026)

From Characters to Tokens: Dynamic Grouping with Hierarchical BPE
par: Dolga, Rares, et autres
Publié: (2025)

AdaptBPE: From General Purpose to Specialized Tokenizers
par: Liyanage, Vijini, et autres
Publié: (2026)

Evaluating Morphological Alignment of Tokenizers in 70 Languages
par: Arnett, Catherine, et autres
Publié: (2025)

MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
par: Asgari, Ehsaneddin, et autres
Publié: (2025)

Even Small Reasoners Should Quote Their Sources: Introducing the Pleias-RAG Model Family
par: Langlais, Pierre-Carl, et autres
Publié: (2025)

Disaggregation Reveals Hidden Training Dynamics: The Case of Agreement Attraction
par: Michaelov, James A., et autres
Publié: (2025)

Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
par: Hayase, Jonathan, et autres
Publié: (2024)

Weight Tying Biases Token Embeddings Towards the Output Space
par: Lopardo, Antonio, et autres
Publié: (2026)

Explaining and Mitigating Crosslingual Tokenizer Inequities
par: Arnett, Catherine, et autres
Publié: (2025)

Transfer of Structural Knowledge from Synthetic Languages
par: Budnikov, Mikhail, et autres
Publié: (2025)

Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
par: Patwary, Firoj Ahmmed, et autres
Publié: (2025)

GPUTOK: GPU Accelerated Byte Level BPE Tokenization
par: Kadamba, Venu Gopal, et autres
Publié: (2026)

Fine-Tuning Transformers: Vocabulary Transfer
par: Mosin, Vladislav, et autres
Publié: (2021)

Sui Generis: Large Language Models for Authorship Attribution and Verification in Latin
par: Schmidt, Gleb, et autres
Publié: (2024)

ComicScene154: A Scene Dataset for Comic Analysis
par: Paval, Sandro, et autres
Publié: (2025)

Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement
par: Arnett, Catherine, et autres
Publié: (2024)

Train It and Forget It: Merge Lists are Unnecessary for BPE Inference in Language Models
par: Sawada, Tomohiro, et autres
Publié: (2025)

Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
par: Vemula, Saketh Reddy, et autres
Publié: (2025)

Knowledge Graph Representation for Political Information Sources
par: Osmonova, Tinatin, et autres
Publié: (2024)

Why do language models perform worse for morphologically complex languages?
par: Arnett, Catherine, et autres
Publié: (2024)

Do Data-based Curricula Work?
par: Surkov, Maxim K., et autres
Publié: (2021)