Cover Image

Where is the signal in tokenization space?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Geh, Renato Lui, Zhang, Honghua, Ahmed, Kareem, Wang, Benjie, Broeck, Guy Van den
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Adversarial Tokenization
by: Geh, Renato Lui, et al.
Published: (2025)

TRACE Back from the Future: A Probabilistic Reasoning Approach to Controllable Language Generation
by: Weng, Gwen Yidou, et al.
Published: (2025)

Controllable Generation via Locally Constrained Resampling
by: Ahmed, Kareem, et al.
Published: (2024)

A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints
by: Ahmed, Kareem, et al.
Published: (2023)

Restructuring Tractable Probabilistic Circuits
by: Zhang, Honghua, et al.
Published: (2024)

On the Relationship Between Monotone and Squared Probabilistic Circuits
by: Wang, Benjie, et al.
Published: (2024)

Probabilistic Programs of Thought
by: Garg, Poorva, et al.
Published: (2026)

Scaling Probabilistic Circuits via Monarch Matrices
by: Zhang, Honghua, et al.
Published: (2025)

How to Marginalize in Causal Structure Learning?
by: Zhao, William, et al.
Published: (2025)

Enabling Autoregressive Models to Fill In Masked Tokens
by: Israel, Daniel, et al.
Published: (2025)

The Pitfalls of KV Cache Compression
by: Chen, Alex, et al.
Published: (2025)

Scaling Tractable Probabilistic Circuits: A Systems Perspective
by: Liu, Anji, et al.
Published: (2024)

Accelerating Diffusion LLMs via Adaptive Parallel Decoding
by: Israel, Daniel, et al.
Published: (2025)

Scaling Up Probabilistic Circuits by Latent Variable Distillation
by: Liu, Anji, et al.
Published: (2022)

Prepacking: A Simple Method for Fast Prefilling and Increased Throughput in Large Language Models
by: Zhao, Siyan, et al.
Published: (2024)

A Compositional Atlas for Algebraic Circuits
by: Wang, Benjie, et al.
Published: (2024)

Tractable Transformers for Flexible Conditional Generation
by: Liu, Anji, et al.
Published: (2025)

SIMPLE: A Gradient Estimator for $k$-Subset Sampling
by: Ahmed, Kareem, et al.
Published: (2022)

Probabilistic Circuits for Cumulative Distribution Functions
by: Broadrick, Oliver, et al.
Published: (2024)

Entropy-Aligned Decoding of LMs for Better Writing and Reasoning
by: Ahmed, Kareem, et al.
Published: (2026)

Adaptable Logical Control for Large Language Models
by: Zhang, Honghua, et al.
Published: (2024)

Probabilistically Rewired Message-Passing Neural Networks
by: Qian, Chendi, et al.
Published: (2023)

Breaking the Factorization Barrier in Diffusion Language Models
by: Li, Ian, et al.
Published: (2026)

ExplainFuzz: Explainable and Constraint-Conditioned Test Generation with Probabilistic Circuits
by: Baiget, Annaëlle, et al.
Published: (2026)

Mitigating Bias in Locally Constrained Decoding via Tractable Proposals
by: Dang, Meihua, et al.
Published: (2026)

Image Inpainting via Tractable Steering of Diffusion Models
by: Liu, Anji, et al.
Published: (2023)

On multi-token prediction for efficient LLM inference
by: Mehra, Somesh, et al.
Published: (2025)

Byte-token Enhanced Language Models for Temporal Point Processes Analysis
by: Kong, Quyu, et al.
Published: (2025)

Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs
by: Singh, Aaditya K., et al.
Published: (2024)

Do language models plan ahead for future tokens?
by: Wu, Wilson, et al.
Published: (2024)

Visualizing token importance for black-box language models
by: Rauba, Paulius, et al.
Published: (2025)

The pitfalls of next-token prediction
by: Bachmann, Gregor, et al.
Published: (2024)

Looking beyond the next token
by: Thankaraj, Abitha, et al.
Published: (2025)

Collapsed Inference for Bayesian Deep Learning
by: Zeng, Zhe, et al.
Published: (2023)

Learning Tractable Distributions Of Language Model Continuations
by: Yidou-Weng, Gwen, et al.
Published: (2025)

Shaping capabilities with token-level data filtering
by: Rathi, Neil, et al.
Published: (2026)

Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations
by: Zhao, Yize, et al.
Published: (2024)

Scaling Transformer to 1M tokens and beyond with RMT
by: Bulatov, Aydar, et al.
Published: (2023)

Zero-Variance Gradients for Variational Autoencoders
by: Shao, Zilei, et al.
Published: (2025)

Rethinking Probabilistic Circuit Parameter Learning
by: Liu, Anji, et al.
Published: (2025)