Cover Image

Splintering Nonconcatenative Languages for Better Tokenization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Gazit, Bar, Shmidman, Shaltiel, Shmidman, Avi, Pinter, Yuval
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

NeoDictaBERT: Pushing the Frontier of BERT models for Hebrew
by: Shmidman, Shaltiel, et al.
Published: (2025)

MRL Parsing Without Tears: The Case of Hebrew
by: Shmidman, Shaltiel, et al.
Published: (2024)

Dicta-LM 3.0: Advancing The Frontier of Hebrew Sovereign LLMs
by: Shmidman, Shaltiel, et al.
Published: (2026)

Adapting LLMs to Hebrew: Unveiling DictaLM 2.0 with Enhanced Vocabulary and Instruction Capabilities
by: Shmidman, Shaltiel, et al.
Published: (2024)

Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?
by: Shmidman, Avi, et al.
Published: (2024)

Learning to Reason: Training LLMs with GPT-OSS or DeepSeek R1 Reasoning Traces
by: Shmidman, Shaltiel, et al.
Published: (2025)

Diversity Over Quantity: A Lesson From Few Shot Relation Classification
by: Cohen, Amir DN, et al.
Published: (2024)

MsBERT: A New Model for the Reconstruction of Lacunae in Hebrew Manuscripts
by: Shmidman, Avi, et al.
Published: (2025)

HEBATRON: A Hebrew-Specialized Open-Weight Mixture-of-Experts Language Model
by: Kayzer, Noam, et al.
Published: (2026)

CharBench: Evaluating the Role of Tokenization in Character-Level Tasks
by: Uzan, Omri, et al.
Published: (2025)

Which Pieces Does Unigram Tokenization Really Need?
by: Land, Sander, et al.
Published: (2025)

Token-Level Privacy in Large Language Models
by: Harel, Re'em, et al.
Published: (2025)

Faster Superword Tokenization
by: Schmidt, Craig W., et al.
Published: (2026)

The Degree of Language Diacriticity and Its Effect on Tasks
by: Cohen, Adi, et al.
Published: (2026)

Protecting Privacy in Classifiers by Token Manipulation
by: Harel, Re'em, et al.
Published: (2024)

Greed is All You Need: An Evaluation of Tokenizer Inference Methods
by: Uzan, Omri, et al.
Published: (2024)

Probing Subphonemes in Morphology Models
by: Astrach, Gal, et al.
Published: (2025)

Hebrew Diacritics Restoration using Visual Representation
by: Elboher, Yair, et al.
Published: (2025)

Information Types in Product Reviews
by: Shapira, Ori, et al.
Published: (2025)

Don't Touch My Diacritics
by: Gorman, Kyle, et al.
Published: (2024)

BiVert: Bidirectional Vocabulary Evaluation using Relations for Machine Translation
by: Cherf, Carinne, et al.
Published: (2024)

How Much is Enough? The Diminishing Returns of Tokenization Training Data
by: Reddy, Varshini, et al.
Published: (2025)

Tokenization with Split Trees
by: Schmidt, Craig W., et al.
Published: (2026)

Tokenization Matters: Navigating Data-Scarce Tokenization for Gender Inclusive Language Technologies
by: Ovalle, Anaelia, et al.
Published: (2023)

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge
by: Batsuren, Khuyagbaatar, et al.
Published: (2024)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
by: Yang, Jinbiao
Published: (2024)

Tokenization Is More Than Compression
by: Schmidt, Craig W., et al.
Published: (2024)

An Analysis of BPE Vocabulary Trimming in Neural Machine Translation
by: Cognetta, Marco, et al.
Published: (2024)

Training a Bilingual Language Model by Mapping Tokens onto a Shared Character Space
by: Rom, Aviad, et al.
Published: (2024)

Better Language Model Inversion by Compactly Representing Next-Token Distributions
by: Nazir, Murtaza, et al.
Published: (2025)

The Effect of Scripts and Formats on LLM Numeracy
by: Reddy, Varshini, et al.
Published: (2026)

Learning to Insert [PAUSE] Tokens for Better Reasoning
by: Kim, Eunki, et al.
Published: (2025)

Broken-Token: Filtering Obfuscated Prompts by Counting Characters-Per-Token
by: Zychlinski, Shaked, et al.
Published: (2025)

Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier
by: Schmidt, Craig W., et al.
Published: (2025)

Silent Tokens, Loud Effects: Padding in LLMs
by: Himelstein, Rom, et al.
Published: (2025)

Byte Latent Transformer: Patches Scale Better Than Tokens
by: Pagnoni, Artidoro, et al.
Published: (2024)

From Tokens to Words: On the Inner Lexicon of LLMs
by: Kaplan, Guy, et al.
Published: (2024)

TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
by: Li, Chong, et al.
Published: (2026)

Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models
by: Kaplan, Guy, et al.
Published: (2025)

OMPar: Automatic Parallelization with AI-Driven Source-to-Source Compilation
by: Kadosh, Tal, et al.
Published: (2024)