Copertina

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Altıntaş, Gül Sena, Ehghaghi, Malikeh, Lester, Brian, Liu, Fengyuan, Zhao, Wanru, Ciccone, Marco, Raffel, Colin
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Computation and Language Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

The Butterfly Effect: Neural Network Training Trajectories Are Highly Sensitive to Initial Conditions
di: Kwok, Devin, et al.
Pubblicazione: (2025)

Arcee's MergeKit: A Toolkit for Merging Large Language Models
di: Goddard, Charles, et al.
Pubblicazione: (2024)

Realistic Evaluation of Model Merging for Compositional Generalization
di: Tam, Derek, et al.
Pubblicazione: (2024)

Zero-Shot Multi-Lingual Speaker Verification in Clinical Trials
di: Akram, Ali, et al.
Pubblicazione: (2024)

Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model
di: Deng, Haikang, et al.
Pubblicazione: (2023)

Merging in a Bottle: Differentiable Adaptive Merging (DAM) and the Path from Averaging to Automation
di: Gauthier-Caron, Thomas, et al.
Pubblicazione: (2024)

ReTok: Replacing Tokenizer to Enhance Representation Efficiency in Large Language Model
di: Gu, Shuhao, et al.
Pubblicazione: (2024)

Merging by Matching Models in Task Parameter Subspaces
di: Tam, Derek, et al.
Pubblicazione: (2023)

TokMem: One-Token Procedural Memory for Large Language Models
di: Wu, Zijun, et al.
Pubblicazione: (2025)

MorphTok: Morphologically Grounded Tokenization for Indian Languages
di: Brahma, Maharaj, et al.
Pubblicazione: (2025)

Position: The Most Expensive Part of an LLM should be its Training Data
di: Kandpal, Nikhil, et al.
Pubblicazione: (2025)

Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models
di: Cho, Gyeongje, et al.
Pubblicazione: (2025)

AttriBoT: A Bag of Tricks for Efficiently Approximating Leave-One-Out Context Attribution
di: Liu, Fengyuan, et al.
Pubblicazione: (2024)

SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
di: Tănase, Andrei-Valentin, et al.
Pubblicazione: (2025)

TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning
di: Zhang, Tunyu, et al.
Pubblicazione: (2025)

FineInstructions: Scaling Synthetic Instructions to Pre-Training Scale
di: Patel, Ajay, et al.
Pubblicazione: (2026)

DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows
di: Patel, Ajay, et al.
Pubblicazione: (2024)

Specialised or Generic? Tokenization Choices for Radiology Language Models
di: Warr, Hermione, et al.
Pubblicazione: (2025)

LoopFormer: Elastic-Depth Looped Transformers for Latent Reasoning via Shortcut Modulation
di: Jeddi, Ahmadreza, et al.
Pubblicazione: (2026)

TokAlign: Efficient Vocabulary Adaptation via Token Alignment
di: Li, Chong, et al.
Pubblicazione: (2025)

Adapting Language Models via Token Translation
di: Feng, Zhili, et al.
Pubblicazione: (2024)

ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization
di: Yadav, Prateek, et al.
Pubblicazione: (2023)

TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment
di: Li, Chong, et al.
Pubblicazione: (2026)

Efficient Model Editing with Task-Localized Sparse Fine-tuning
di: Iurada, Leonardo, et al.
Pubblicazione: (2025)

Towards Universal Semantics With Large Language Models
di: Baartmans, Raymond, et al.
Pubblicazione: (2025)

Problematic Tokens: Tokenizer Bias in Large Language Models
di: Yang, Jin, et al.
Pubblicazione: (2024)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
di: Pan, Bowen, et al.
Pubblicazione: (2024)

S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models
di: Lei, Fangyu, et al.
Pubblicazione: (2023)

The Impact of Token Granularity on the Predictive Power of Language Model Surprisal
di: Oh, Byung-Doh, et al.
Pubblicazione: (2024)

Scaling Data-Constrained Language Models
di: Muennighoff, Niklas, et al.
Pubblicazione: (2023)

Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations
di: Zheng, Brian Siyuan, et al.
Pubblicazione: (2025)

Counting Ability of Large Language Models and Impact of Tokenization
di: Zhang, Xiang, et al.
Pubblicazione: (2024)

The Appeal and Reality of Recycling LoRAs with Adaptive Merging
di: Liu, Haokun, et al.
Pubblicazione: (2026)

Efficiently Estimating Data Efficiency for Language Model Fine-tuning
di: Je, Gyung Hyun, et al.
Pubblicazione: (2025)

MentraSuite: Post-Training Large Language Models for Mental Health Reasoning and Assessment
di: Xiao, Mengxi, et al.
Pubblicazione: (2025)

DynTok: Dynamic Compression of Visual Tokens for Efficient and Effective Video Understanding
di: Zhang, Hongzhi, et al.
Pubblicazione: (2025)

Leveraging Language Models for Emotion and Behavior Analysis in Education
di: Tanaka, Kaito, et al.
Pubblicazione: (2024)

MSTS: A Multimodal Safety Test Suite for Vision-Language Models
di: Röttger, Paul, et al.
Pubblicazione: (2025)

TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation
di: Lin, Haokun, et al.
Pubblicazione: (2025)

Behavior-Equivalent Token: Single-Token Replacement for Long Prompts in LLMs
di: Dong, Jiancheng, et al.
Pubblicazione: (2025)