Cover Image

The Impact of Token Granularity on the Predictive Power of Language Model Surprisal

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Oh, Byung-Doh, Schuler, William
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

The Inverse Scaling Effect of Pre-Trained Language Model Surprisal Is Not Due to Data Leakage
by: Oh, Byung-Doh, et al.
Published: (2025)

Frequency Explains the Inverse Correlation of Large Language Models' Size, Training Data Amount, and Surprisal's Fit to Reading Times
by: Oh, Byung-Doh, et al.
Published: (2024)

Leading Whitespaces of Language Models' Subword Vocabulary Pose a Confound for Calculating Word Probabilities
by: Oh, Byung-Doh, et al.
Published: (2024)

Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal
by: Nair, Sathvik, et al.
Published: (2026)

How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor?
by: Clark, Christian, et al.
Published: (2025)

Linear Recency Bias During Training Improves Transformers' Fit to Reading Times
by: Clark, Christian, et al.
Published: (2024)

Surprisal from Larger Transformer-based Language Models Predicts fMRI Data More Poorly
by: Lin, Yi-Chien, et al.
Published: (2025)

Testing the Predictions of Surprisal Theory in 11 Languages
by: Wilcox, Ethan Gotlieb, et al.
Published: (2023)

Vectors from Larger Language Models Predict Human Reading Time and fMRI Data More Poorly when Dimensionality Expansion is Controlled
by: Lin, Yi-Chien, et al.
Published: (2025)

Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokens
by: Zhang, Anqi, et al.
Published: (2024)

The Frequency Confound in Language-Model Surprisal and Metaphor Novelty
by: Momen, Omar, et al.
Published: (2026)

Language Models are Surprisingly Fragile to Drug Names in Biomedical Benchmarks
by: Gallifant, Jack, et al.
Published: (2024)

Tokenization, Fusion and Decoupling: Bridging the Granularity Mismatch Between Large Language Models and Knowledge Graphs
by: Su, Siyue, et al.
Published: (2026)

Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse
by: Tsipidi, Eleftheria, et al.
Published: (2024)

Parallel Token Prediction for Language Models
by: Draxler, Felix, et al.
Published: (2025)

Confabulation: The Surprising Value of Large Language Model Hallucinations
by: Sui, Peiqi, et al.
Published: (2024)

Counting Ability of Large Language Models and Impact of Tokenization
by: Zhang, Xiang, et al.
Published: (2024)

ByteScience: Bridging Unstructured Scientific Literature and Structured Data with Auto Fine-tuned Large Language Model in Token Granularity
by: Xie, Tong, et al.
Published: (2024)

Contextually Structured Token Dependency Encoding for Large Language Models
by: Blades, James, et al.
Published: (2025)

Addressing Topic Granularity and Hallucination in Large Language Models for Topic Modelling
by: Mu, Yida, et al.
Published: (2024)

On The Truthfulness of 'Surprisingly Likely' Responses of Large Language Models
by: Goel, Naman
Published: (2023)

Multi-Granularity Semantic Revision for Large Language Model Distillation
by: Liu, Xiaoyu, et al.
Published: (2024)

An Existence Proof for Neural Language Models That Can Explain Garden-Path Effects via Surprisal
by: Yoshida, Ryo, et al.
Published: (2026)

Problematic Tokens: Tokenizer Bias in Large Language Models
by: Yang, Jin, et al.
Published: (2024)

Surprisal Minimisation over Goal-directed Alternatives Predicts Production Choice in Dialogue
by: Utting, Tom, et al.
Published: (2026)

Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts
by: Chen, Yingfa, et al.
Published: (2024)

Token Signature: Predicting Chain-of-Thought Gains with Token Decoding Feature in Large Language Models
by: Liu, Peijie, et al.
Published: (2025)

From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Prediction
by: Zhu, Mingcheng, et al.
Published: (2026)

Rethinking Token Prediction: Tree-Structured Diffusion Language Model
by: Wu, Zihao, et al.
Published: (2026)

Efficient Temporal Tokenization for Mobility Prediction with Large Language Models
by: He, Haoyu, et al.
Published: (2025)

Pre-Training Curriculum for Multi-Token Prediction in Language Models
by: Aynetdinov, Ansar, et al.
Published: (2025)

OrthoRank: Token Selection via Sink Token Orthogonality for Efficient LLM inference
by: Shin, Seungjun, et al.
Published: (2025)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
by: Yang, Jinbiao
Published: (2024)

Technical Report: Impact of Position Bias on Language Models in Token Classification
by: Amor, Mehdi Ben, et al.
Published: (2023)

ExLM: Rethinking the Impact of [MASK] Tokens in Masked Language Models
by: Zheng, Kangjie, et al.
Published: (2025)

TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
by: Altıntaş, Gül Sena, et al.
Published: (2025)

MTA: Multi-Granular Trajectory Alignment for Large Language Model Distillation
by: Chi, Pham Khanh, et al.
Published: (2026)

Granular Privacy Control for Geolocation with Vision Language Models
by: Mendes, Ethan, et al.
Published: (2024)

Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMs
by: Oh, Gyutaek, et al.
Published: (2025)

Surprising Efficacy of Fine-Tuned Transformers for Fact-Checking over Larger Language Models
by: Setty, Vinay
Published: (2024)