Cover Image

Are you going to finish that? A Practical Study of the Partial Token Problem

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Xu, Hao, Liu, Alisa, Hayase, Jonathan, Choi, Yejin, Smith, Noah A.
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?
by: Hayase, Jonathan, et al.
Published: (2024)

Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations
by: Zheng, Brian Siyuan, et al.
Published: (2025)

SuperBPE: Space Travel for Language Models
by: Liu, Alisa, et al.
Published: (2025)

Sampling from Your Language Model One Byte at a Time
by: Hayase, Jonathan, et al.
Published: (2025)

Tuning Language Models by Proxy
by: Liu, Alisa, et al.
Published: (2024)

Infini-gram mini: Exact n-gram Search at the Internet Scale with FM-Index
by: Xu, Hao, et al.
Published: (2025)

A Taxonomy of Ambiguity Types for NLP
by: Li, Margaret Y., et al.
Published: (2024)

Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedback
by: Ivison, Hamish, et al.
Published: (2024)

A Call for Clarity in Beam Search: How It Works and When It Stops
by: Kasai, Jungo, et al.
Published: (2022)

Does Liking Yellow Imply Driving a School Bus? Semantic Leakage in Language Models
by: Gonen, Hila, et al.
Published: (2024)

Infini-gram: Scaling Unbounded n-gram Language Models to a Trillion Tokens
by: Liu, Jiacheng, et al.
Published: (2024)

Measuring and Improving Attentiveness to Partial Inputs with Counterfactuals
by: Elazar, Yanai, et al.
Published: (2023)

LLAMAPIE: Proactive In-Ear Conversation Assistants
by: Chen, Tuochao, et al.
Published: (2025)

Opt-ICL at LeWiDi-2025: Maximizing In-Context Signal from Rater Examples via Meta-Learning
by: Sorensen, Taylor, et al.
Published: (2025)

Compute Optimal Tokenization
by: Limisiewicz, Tomasz, et al.
Published: (2026)

Risks and NLP Design: A Case Study on Procedural Document QA
by: Haduong, Nikita, et al.
Published: (2024)

Learning to Compress Prompts with Gist Tokens
by: Mu, Jesse, et al.
Published: (2023)

MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization
by: Ahia, Orevaoghene, et al.
Published: (2024)

RealTime QA: What's the Answer Right Now?
by: Kasai, Jungo, et al.
Published: (2022)

Rewriting History: A Recipe for Interventional Analyses to Study Data Effects on Model Behavior
by: Nadkarni, Rahul, et al.
Published: (2025)

Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement
by: Jung, Jaehun, et al.
Published: (2024)

Can Language Models Reason about Individualistic Human Values and Preferences?
by: Jiang, Liwei, et al.
Published: (2024)

Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs
by: Wang, Siyuan, et al.
Published: (2024)

Symbolic Working Memory Enhances Language Models for Complex Rule Application
by: Wang, Siyuan, et al.
Published: (2024)

Safeguarding RAG Pipelines with GMTP: A Gradient-based Masked Token Probability Method for Poisoned Document Detection
by: Kim, San, et al.
Published: (2025)

Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model
by: He, Jacqueline, et al.
Published: (2026)

Language Model Alignment in Multilingual Trolley Problems
by: Jin, Zhijing, et al.
Published: (2024)

Op-Fed: Opinion, Stance, and Monetary Policy Annotations on FOMC Transcripts Using Active Learning
by: Kanganis, Alisa, et al.
Published: (2025)

OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens
by: Liu, Jiacheng, et al.
Published: (2025)

From Dogwhistles to Bullhorns: Unveiling Coded Rhetoric with Language Models
by: Mendelsohn, Julia, et al.
Published: (2023)

Think before you speak: Training Language Models With Pause Tokens
by: Goyal, Sachin, et al.
Published: (2023)

From Explicit CoT to Implicit CoT: Learning to Internalize CoT Step by Step
by: Deng, Yuntian, et al.
Published: (2024)

Understanding Dataset Difficulty with $\mathcal{V}$-Usable Information
by: Ethayarajh, Kawin, et al.
Published: (2021)

Reading Between the Lines: The One-Sided Conversation Problem
by: Ebert, Victoria, et al.
Published: (2025)

DiffAdapt: Difficulty-Adaptive Reasoning for Token-Efficient LLM Inference
by: Liu, Xiang, et al.
Published: (2025)

When One LLM Drools, Multi-LLM Collaboration Rules
by: Feng, Shangbin, et al.
Published: (2025)

Sample, Don't Search: Rethinking Test-Time Alignment for Language Models
by: Faria, Gonçalo, et al.
Published: (2025)

Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild
by: Mireshghallah, Niloofar, et al.
Published: (2024)

Query-Based Adversarial Prompt Generation
by: Hayase, Jonathan, et al.
Published: (2024)

HALoGEN: Fantastic LLM Hallucinations and Where to Find Them
by: Ravichander, Abhilasha, et al.
Published: (2025)