Cover Image

From Tokens to Words: On the Inner Lexicon of LLMs

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kaplan, Guy, Oren, Matanel, Reif, Yuval, Schwartz, Roy
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic
by: Reif, Yuval, et al.
Published: (2025)

Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models
by: Kaplan, Guy, et al.
Published: (2025)

Beyond Performance: Quantifying and Mitigating Label Bias in LLMs
by: Reif, Yuval, et al.
Published: (2024)

Why Fine-Tuning Encourages Hallucinations and How to Fix It
by: Kaplan, Guy, et al.
Published: (2026)

Transformers are Multi-State RNNs
by: Oren, Matanel, et al.
Published: (2024)

Do LLMs Benefit From Their Own Words?
by: Huang, Jenny Y., et al.
Published: (2026)

From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Test
by: Dai, Xunlian, et al.
Published: (2025)

The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
by: Thakur, Aamod, et al.
Published: (2025)

Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models
by: Cho, Gyeongje, et al.
Published: (2025)

Evaluating Lexicon Incorporation for Depression Symptom Estimation
by: Milintsevich, Kirill, et al.
Published: (2024)

Broken-Token: Filtering Obfuscated Prompts by Counting Characters-Per-Token
by: Zychlinski, Shaked, et al.
Published: (2025)

From Words to Proverbs: Evaluating LLMs Linguistic and Cultural Competence in Saudi Dialects with Absher
by: Al-Monef, Renad, et al.
Published: (2025)

Words that make SENSE: Sensorimotor Norms in Learned Lexical Token Representations
by: Gupta, Abhinav, et al.
Published: (2026)

From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages
by: Kiulian, Artur, et al.
Published: (2024)

From Implicit to Explicit: Token-Efficient Logical Supervision for Mathematical Reasoning in LLMs
by: Wang, Shaojie, et al.
Published: (2026)

Facts Do Care About Your Language: Assessing Answer Quality of Multilingual LLMs
by: Kansal, Yuval, et al.
Published: (2025)

LLM Factoscope: Uncovering LLMs' Factual Discernment through Inner States Analysis
by: He, Jinwen, et al.
Published: (2023)

From Next-Token to Next-Block: A Principled Adaptation Path for Diffusion LLMs
by: Tian, Yuchuan, et al.
Published: (2025)

Adversarial Math Word Problem Generation
by: Xie, Roy, et al.
Published: (2024)

LLMs are Not Just Next Token Predictors
by: Downes, Stephen M., et al.
Published: (2024)

Enhanced Sentiment Interpretation via a Lexicon-Fuzzy-Transformer Framework
by: Rokhva, Shayan, et al.
Published: (2025)

Word Form Matters: LLMs' Semantic Reconstruction under Typoglycemia
by: Wang, Chenxi, et al.
Published: (2025)

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge
by: Batsuren, Khuyagbaatar, et al.
Published: (2024)

TECP: Token-Entropy Conformal Prediction for LLMs
by: Xu, Beining, et al.
Published: (2025)

Visual Editing with LLM-based Tool Chaining: An Efficient Distillation Approach for Real-Time Applications
by: Sultan, Oren, et al.
Published: (2024)

Breaking the Cloak! Unveiling Chinese Cloaked Toxicity with Homophone Graph and Toxic Lexicon
by: Ma, Xuchen, et al.
Published: (2025)

Beyond Words: A Latent Memory Approach to Internal Reasoning in LLMs
by: Orlicki, José I.
Published: (2025)

Caught in the Web of Words: Do LLMs Fall for Spin in Medical Literature?
by: Yun, Hye Sun, et al.
Published: (2025)

Adversarial Tokenization
by: Geh, Renato Lui, et al.
Published: (2025)

Say Anything but This: When Tokenizer Betrays Reasoning in LLMs
by: Ayoobi, Navid, et al.
Published: (2026)

TokenSkip: Controllable Chain-of-Thought Compression in LLMs
by: Xia, Heming, et al.
Published: (2025)

Discrete Tokenization for Multimodal LLMs: A Comprehensive Survey
by: Li, Jindong, et al.
Published: (2025)

From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning
by: Shani, Chen, et al.
Published: (2025)

Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning
by: Hassid, Michael, et al.
Published: (2025)

PhiloBERTA: A Transformer-Based Cross-Lingual Analysis of Greek and Latin Lexicons
by: Allbert, Rumi, et al.
Published: (2025)

An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing
by: Chai, Ziwei, et al.
Published: (2024)

Psycholinguistic Word Features: a New Approach for the Evaluation of LLMs Alignment with Humans
by: Conde, Javier, et al.
Published: (2025)

From Word Sequences to Behavioral Sequences: Adapting Modeling and Evaluation Paradigms for Longitudinal NLP
by: Ganesan, Adithya V, et al.
Published: (2026)

Phonetic Perturbations Reveal Tokenizer-Rooted Safety Gaps in LLMs
by: Aswal, Darpan, et al.
Published: (2025)

Hessian-Enhanced Token Attribution (HETA): Interpreting Autoregressive LLMs
by: Pramanik, Vishal, et al.
Published: (2026)