Imagen de Portada

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Altıntaş, Gül Sena, Ehghaghi, Malikeh, Lester, Brian, Liu, Fengyuan, Zhao, Wanru, Ciccone, Marco, Raffel, Colin
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions
por: Kwok, Devin, et al.
Publicado: (2025)

Arcee's MergeKit: A Toolkit for Merging Large Language Models
por: Goddard, Charles, et al.
Publicado: (2024)

Realistic Evaluation of Model Merging for Compositional Generalization
por: Tam, Derek, et al.
Publicado: (2024)

Zero-Shot Multi-Lingual Speaker Verification in Clinical Trials
por: Akram, Ali, et al.
Publicado: (2024)

Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
por: Deng, Haikang, et al.
Publicado: (2023)

Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automation
por: Gauthier-Caron, Thomas, et al.
Publicado: (2024)

ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Model
por: Gu, Shuhao, et al.
Publicado: (2024)

Merging by Matching Models in Task Parameter Subspaces
por: Tam, Derek, et al.
Publicado: (2023)

TokMem: One-Token Procedural Memory for Large Language Models
por: Wu, Zijun, et al.
Publicado: (2025)

MorphTok: Morphologically Grounded Tokenization for Indian Languages
por: Brahma, Maharaj, et al.
Publicado: (2025)

Position: The Most Expensive Part of an LLM should be its Training Data
por: Kandpal, Nikhil, et al.
Publicado: (2025)

Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models
por: Cho, Gyeongje, et al.
Publicado: (2025)

AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution
por: Liu, Fengyuan, et al.
Publicado: (2024)

SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
por: Tănase, Andrei-Valentin, et al.
Publicado: (2025)

TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning
por: Zhang, Tunyu, et al.
Publicado: (2025)

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
por: Patel, Ajay, et al.
Publicado: (2026)

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows
por: Patel, Ajay, et al.
Publicado: (2024)

Specialised or Generic? Tokenization Choices for Radiology Language Models
por: Warr, Hermione, et al.
Publicado: (2025)

LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
por: Jeddi, Ahmadreza, et al.
Publicado: (2026)

TokAlign: Efficient Vocabulary Adaptation via Token Alignment
por: Li, Chong, et al.
Publicado: (2025)

Adapting Language Models via Token Translation
por: Feng, Zhili, et al.
Publicado: (2024)

ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
por: Yadav, Prateek, et al.
Publicado: (2023)

TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
por: Li, Chong, et al.
Publicado: (2026)

Efficient Model Editing with Task-Localized Sparse Fine-tuning
por: Iurada, Leonardo, et al.
Publicado: (2025)

Towards Universal Semantics With Large Language Models
por: Baartmans, Raymond, et al.
Publicado: (2025)

Problematic Tokens: Tokenizer Bias in Large Language Models
por: Yang, Jin, et al.
Publicado: (2024)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
por: Pan, Bowen, et al.
Publicado: (2024)

S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models
por: Lei, Fangyu, et al.
Publicado: (2023)

The Impact of Token Granularity on the Predictive Power of Language Model Surprisal
por: Oh, Byung-Doh, et al.
Publicado: (2024)

Scaling Data-Constrained Language Models
por: Muennighoff, Niklas, et al.
Publicado: (2023)

Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations
por: Zheng, Brian Siyuan, et al.
Publicado: (2025)

Counting Ability of Large Language Models and Impact of Tokenization
por: Zhang, Xiang, et al.
Publicado: (2024)

The Appeal and Reality of Recycling LoRAs with Adaptive Merging
por: Liu, Haokun, et al.
Publicado: (2026)

Efficiently Estimating Data Efficiency for Language Model Fine-tuning
por: Je, Gyung Hyun, et al.
Publicado: (2025)

MentraSuite: Post-Training Large Language Models for Mental Health Reasoning and Assessment
por: Xiao, Mengxi, et al.
Publicado: (2025)

DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding
por: Zhang, Hongzhi, et al.
Publicado: (2025)

Leveraging Language Models for Emotion and Behavior Analysis in Education
por: Tanaka, Kaito, et al.
Publicado: (2024)

MSTS: A Multimodal Safety Test Suite for Vision-Language Models
por: Röttger, Paul, et al.
Publicado: (2025)

TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation
por: Lin, Haokun, et al.
Publicado: (2025)

Behavior-Equivalent Token: Single-Token Replacement for Long Prompts in LLMs
por: Dong, Jiancheng, et al.
Publicado: (2025)