Cover Image

Is There a Case for Conversation Optimized Tokenizers in Large Language Models?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ferrando, Raquel, Conde, Javier, Martínez, Gonzalo, Reviriego, Pedro
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Speed and Conversational Large Language Models: Not All Is About Tokens per Second
by: Conde, Javier, et al.
Published: (2025)

The Generative Energy Arena (GEA): Incorporating Energy Awareness in Large Language Model (LLM) Human Evaluations
by: Arriaga, Carlos, et al.
Published: (2025)

Multiple Choice Questions: Reasoning Makes Large Language Models (LLMs) More Self-Confident, Especially When They are Wrong
by: Fu, Tairan, et al.
Published: (2025)

Psycholinguistic Word Features: a New Approach for the Evaluation of LLMs Alignment with Humans
by: Conde, Javier, et al.
Published: (2025)

Why Do Large Language Models (LLMs) Struggle to Count Letters?
by: Fu, Tairan, et al.
Published: (2024)

Can ChatGPT Learn to Count Letters?
by: Conde, Javier, et al.
Published: (2025)

Playing with words: Comparing the vocabulary and lexical diversity of ChatGPT and humans
by: Reviriego, Pedro, et al.
Published: (2023)

Concurrent Linguistic Error Detection (CLED): a New Methodology for Error Detection in Large Language Models
by: Zhu, Jinhua, et al.
Published: (2024)

How does fine-tuning improve sensorimotor representations in large language models?
by: Wu, Minghua, et al.
Published: (2026)

Beyond Reproducibility: Token Probabilities Expose Large Language Model Nondeterminism
by: Fu, Tairan, et al.
Published: (2026)

Lost in Sampling: Assessing Lexical Reachability in LLMs via the Word Coverage Score (WCS)
by: Awad, Samer, et al.
Published: (2026)

Information Flow Routes: Automatically Interpreting Language Models at Scale
by: Ferrando, Javier, et al.
Published: (2024)

Beware of Words: Evaluating the Lexical Diversity of Conversational LLMs using ChatGPT as Case Study
by: Martínez, Gonzalo, et al.
Published: (2024)

Open Conversational LLMs do not know most Spanish words
by: Conde, Javier, et al.
Published: (2024)

To Words and Beyond: Probing Large Language Models for Sentence-Level Psycholinguistic Norms of Memorability and Reading Times
by: Clark, Thomas Hikaru, et al.
Published: (2026)

Automated Interpretability and Feature Discovery in Language Models with Agents
by: Marin-Llobet, Arnau, et al.
Published: (2026)

Spanish and LLM Benchmarks: is MMLU Lost in Translation?
by: Plaza, Irene, et al.
Published: (2024)

CROP: Token-Efficient Reasoning in Large Language Models via Regularized Prompt Optimization
by: Shah, Deep, et al.
Published: (2026)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
by: Wang, Dixuan, et al.
Published: (2024)

findsylls: A Language-Agnostic Toolkit for Syllable-Level Speech Tokenization and Embedding
by: Martínez, Héctor Javier Vázquez
Published: (2026)

Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Models
by: Ferrando, Javier, et al.
Published: (2024)

Counting Ability of Large Language Models and Impact of Tokenization
by: Zhang, Xiang, et al.
Published: (2024)

Establishing Vocabulary Tests as a Benchmark for Evaluating Large Language Models
by: Martínez, Gonzalo, et al.
Published: (2023)

Optimizing Language Augmentation for Multilingual Large Language Models: A Case Study on Korean
by: Choi, ChangSu, et al.
Published: (2024)

TLPO: Token-Level Policy Optimization for Mitigating Language Confusion in Large Language Models
by: Choo, Jinho, et al.
Published: (2026)

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
by: Liu, Peijie, et al.
Published: (2025)

Large Language Models for Automating Clinical Data Standardization: HL7 FHIR Use Case
by: Riquelme, Alvaro, et al.
Published: (2025)

Contextual Reinforcement in Multimodal Token Compression for Large Language Models
by: Piero, Naderdel, et al.
Published: (2025)

Saliency-driven Dynamic Token Pruning for Large Language Models
by: Tao, Yao, et al.
Published: (2025)

How Confident Is the First Token? An Uncertainty-Calibrated Prompt Optimization Framework for Large Language Model Classification and Understanding
by: Chen, Wei, et al.
Published: (2026)

Evaluating Tokenizer Performance of Large Language Models Across Official Indian Languages
by: Tamang, S., et al.
Published: (2024)

Large Language Models as Misleading Assistants in Conversation
by: Hou, Betty Li, et al.
Published: (2024)

Re-Initialization Token Learning for Tool-Augmented Large Language Models
by: Li, Chenghao, et al.
Published: (2025)

Memory Retrieval and Consolidation in Large Language Models through Function Tokens
by: Zhang, Shaohua, et al.
Published: (2025)

Enhancing Large Language Models for Mobility Analytics with Semantic Location Tokenization
by: Chen, Yile, et al.
Published: (2025)

TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models
by: Yang, Zhen, et al.
Published: (2023)

Cross-Lingual Conversational Speech Summarization with Large Language Models
by: Nelson, Max, et al.
Published: (2024)

Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models
by: Castillo-Bolado, David, et al.
Published: (2024)

Direct Token Optimization: A Self-contained Approach to Large Language Model Unlearning
by: Lee, Hong kyu, et al.
Published: (2025)

Counterfactual Token Generation in Large Language Models
by: Chatzi, Ivi, et al.
Published: (2024)