Image de couverture de livre

Token Alignment via Character Matching for Subword Completion

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Athiwaratkun, Ben, Wang, Shiqi, Shang, Mingyue, Tian, Yuchen, Wang, Zijian, Gonugondla, Sujan Kumar, Gouda, Sanjay Krishna, Kwiatowski, Rob, Nallapati, Ramesh, Xiang, Bing
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

BASS: Batched Attention-optimized Speculative Sampling
par: Qian, Haifeng, et autres
Publié: (2024)

Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs
par: Athiwaratkun, Ben, et autres
Publié: (2024)

CodeFort: Robust Training for Code Generation Models
par: Zhang, Yuhao, et autres
Publié: (2024)

Learning Mutually Informed Representations for Characters and Subwords
par: Wang, Yilin, et autres
Publié: (2023)

The N-Grammys: Accelerating Autoregressive Inference with Learning-Free Batched Speculation
par: Stewart, Lawrence, et autres
Publié: (2024)

Scaling Instruction-Tuned LLMs to Million-Token Contexts via Hierarchical Synthetic Data Generation
par: He, Linda, et autres
Publié: (2025)

Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks
par: Saha, Shoumik, et autres
Publié: (2025)

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge
par: Batsuren, Khuyagbaatar, et autres
Publié: (2024)

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features
par: Stephen, Abishek, et autres
Publié: (2026)

Subword Tokenization Strategies for Kurdish Word Embeddings
par: Salehi, Ali, et autres
Publié: (2025)

Reasoning in Token Economies: Budget-Aware Evaluation of LLM Reasoning Strategies
par: Wang, Junlin, et autres
Publié: (2024)

Constrained Decoding for Fill-in-the-Middle Code Language Models via Efficient Left and Right Quotienting of Context-Sensitive Grammars
par: Melcer, Daniel, et autres
Publié: (2024)

Chinese Word Boundary Recovery through Character Alignment Projection
par: Wang, Lusha, et autres
Publié: (2026)

Subwords as Skills: Tokenization for Sparse-Reward Reinforcement Learning
par: Yunis, David, et autres
Publié: (2023)

Improving Model Alignment Through Collective Intelligence of Open-Source LLMS
par: Wang, Junlin, et autres
Publié: (2025)

Code Representation Learning At Scale
par: Zhang, Dejiao, et autres
Publié: (2024)

Tokenization Falling Short: On Subword Robustness in Large Language Models
par: Chai, Yekun, et autres
Publié: (2024)

Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM
par: Ryan, Gabriel, et autres
Publié: (2024)

Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenization
par: Li, Zilong
Publié: (2024)

Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT
par: Wolleb, Benoist, et autres
Publié: (2023)

Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
par: Patwary, Firoj Ahmmed, et autres
Publié: (2025)

SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers
par: Chelombitko, Iaroslav, et autres
Publié: (2026)

Data Diversification Methods In Alignment Enhance Math Performance In LLMs
par: Dokmeci, Berkan, et autres
Publié: (2025)

Mixture-of-Agents Enhances Large Language Model Capabilities
par: Wang, Junlin, et autres
Publié: (2024)

When Does Divide and Conquer Work for Long Context LLM? A Noise Decomposition Framework
par: Xu, Zhen, et autres
Publié: (2025)

Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods
par: Suyunu, Burak, et autres
Publié: (2024)

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
par: Gigant, Théo, et autres
Publié: (2026)

Can Pretrained Language Models Derive Correct Semantics from Corrupt Subwords under Noise?
par: Li, Xinzhe, et autres
Publié: (2023)

T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings
par: Deiseroth, Björn, et autres
Publié: (2024)

Tokenization Disparities as Infrastructure Bias: How Subword Systems Create Inequities in LLM Access and Efficiency
par: Teklehaymanot, Hailay Kidu, et autres
Publié: (2025)

From Smør-re-brød to Subwords: Training LLMs on Danish, One Morpheme at a Time
par: Kildeberg, Mikkel Wildner, et autres
Publié: (2025)

MOSAIC: Composable Safety Alignment with Modular Control Tokens
par: Peng, Jingyu, et autres
Publié: (2026)

Approximately Aligned Decoding
par: Melcer, Daniel, et autres
Publié: (2024)

Distributional Properties of Subword Regularization
par: Cognetta, Marco, et autres
Publié: (2024)

Lexically Grounded Subword Segmentation
par: Libovický, Jindřich, et autres
Publié: (2024)

Random Subwords and Pipe Dreams
par: Defant, Colin
Publié: (2024)

MTVCraft: Tokenizing 4D Motion for Arbitrary Character Animation
par: Ding, Yanbo, et autres
Publié: (2025)

Existential Definability over the Subword Ordering
par: Baumann, Pascal, et autres
Publié: (2022)

On the Partial Sum of Subword-Counting Sequences
par: Jain, Pranjal, et autres
Publié: (2024)

TTPA: Token-level Tool-use Preference Alignment Training Framework with Fine-grained Evaluation
par: Huang, Chengrui, et autres
Publié: (2025)