Cover Image

Retrofitting Large Language Models with Dynamic Tokenization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Feher, Darius, Vulić, Ivan, Minixhofer, Benjamin
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Zero-Shot Tokenizer Transfer
by: Minixhofer, Benjamin, et al.
Published: (2024)

Universal Cross-Tokenizer Distillation via Approximate Likelihood Matching
by: Minixhofer, Benjamin, et al.
Published: (2025)

Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentation
by: Frohmann, Markus, et al.
Published: (2024)

Quantifying Language Disparities in Multilingual Large Language Models
by: Hu, Songbo, et al.
Published: (2025)

Analyzing and Adapting Large Language Models for Few-Shot Multilingual NLU: Are We There Yet?
by: Razumovskaia, Evgeniia, et al.
Published: (2024)

On Bilingual Lexicon Induction with Large Language Models
by: Li, Yaoyiran, et al.
Published: (2023)

Specialising and Analysing Instruction-Tuned and Byte-Level Language Models for Organic Reaction Prediction
by: Pang, Jiayun, et al.
Published: (2024)

DIALIGHT: Lightweight Multilingual Development and Evaluation of Task-Oriented Dialogue Systems with Large Language Models
by: Hu, Songbo, et al.
Published: (2024)

Large Language Models are Miscalibrated In-Context Learners
by: Li, Chengzu, et al.
Published: (2023)

Polyglot Teachers: Evaluating Language Models for Multilingual Synthetic Data Generation
by: Miranda, Lester James V., et al.
Published: (2026)

Navigating the Alignment-Calibration Trade-off: A Pareto-Superior Frontier via Model Merging
by: Hu, Tiancheng, et al.
Published: (2025)

UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models
by: Dong, Yijiang River, et al.
Published: (2024)

Scaling Sparse Fine-Tuning to Large Language Models
by: Ansell, Alan, et al.
Published: (2024)

Self-Distillation for Model Stacking Unlocks Cross-Lingual NLU in 200+ Languages
by: Schmidt, Fabian David, et al.
Published: (2024)

Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models
by: Zhou, Ej, et al.
Published: (2025)

Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models
by: Liu, Yinhong, et al.
Published: (2024)

ReCoVeR the Target Language: Language Steering without Sacrificing Task Performance
by: Sterz, Hannah, et al.
Published: (2025)

Fairer Preferences Elicit Improved Human-Aligned Large Language Model Judgments
by: Zhou, Han, et al.
Published: (2024)

Dynamic Memory Compression: Retrofitting LLMs for Accelerated Inference
by: Nawrot, Piotr, et al.
Published: (2024)

DARE: Diverse Visual Question Answering with Robustness Evaluation
by: Sterz, Hannah, et al.
Published: (2024)

Language Fusion for Parameter-Efficient Cross-lingual Transfer
by: Borchert, Philipp, et al.
Published: (2025)

Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluators
by: Liu, Yinhong, et al.
Published: (2024)

Lost in Embeddings: Information Loss in Vision-Language Models
by: Li, Wenyan, et al.
Published: (2025)

Problematic Tokens: Tokenizer Bias in Large Language Models
by: Yang, Jin, et al.
Published: (2024)

Bolmo: Byteifying the Next Generation of Language Models
by: Minixhofer, Benjamin, et al.
Published: (2025)

Learning to Generate and Evaluate Fact-checking Explanations with Transformers
by: Feher, Darius, et al.
Published: (2024)

Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
by: McLeish, Sean, et al.
Published: (2025)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
by: Yang, Jinbiao
Published: (2024)

Saliency-driven Dynamic Token Pruning for Large Language Models
by: Tao, Yao, et al.
Published: (2025)

SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU
by: Razumovskaia, Evgeniia, et al.
Published: (2023)

Fleurs-SLU: A Massively Multilingual Benchmark for Spoken Language Understanding
by: Schmidt, Fabian David, et al.
Published: (2025)

Large Language Model as Token Compressor and Decompressor
by: Li, Wenbing, et al.
Published: (2026)

TopViewRS: Vision-Language Models as Top-View Spatial Reasoners
by: Li, Chengzu, et al.
Published: (2024)

Why do Large Language Models Fail in Low-resource Translation? Unraveling the Token Dynamics of Large Language Models for Machine Translation
by: Qian, Shenbin, et al.
Published: (2026)

Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language
by: Tamang, Sagar, et al.
Published: (2024)

FUN with Fisher: Improving Generalization of Adapter-Based Cross-lingual Transfer with Scheduled Unfreezing
by: Liu, Chen Cecilia, et al.
Published: (2023)

Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Models
by: Vazhentsev, Artem, et al.
Published: (2025)

TTSDS2: Resources and Benchmark for Evaluating Human-Quality Text to Speech Systems
by: Minixhofer, Christoph, et al.
Published: (2025)

Rethinking Personalization in Large Language Models at the Token Level
by: Zhang, Chenheng, et al.
Published: (2026)

Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models
by: Chang, Tyler A., et al.
Published: (2025)