Cover Image

From Language Models over Tokens to Language Models over Characters

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Vieira, Tim, LeBrun, Ben, Giulianelli, Mario, Gastaldi, Juan Luis, DuSell, Brian, Terilla, John, O'Donnell, Timothy J., Cotterell, Ryan
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Language Models over Canonical Byte-Pair Encodings
by: Vieira, Tim, et al.
Published: (2025)

The Foundations of Tokenization: Statistical and Computational Concerns
by: Gastaldi, Juan Luis, et al.
Published: (2024)

On the Proper Treatment of Tokenization in Psycholinguistics
by: Giulianelli, Mario, et al.
Published: (2024)

Information Locality as an Inductive Bias for Neural Language Models
by: Someya, Taiga, et al.
Published: (2025)

Bearing Syntactic Fruit with Stack-Augmented Neural Networks
by: DuSell, Brian, et al.
Published: (2025)

Algorithms for Weighted Pushdown Automata
by: Butoi, Alexandra, et al.
Published: (2022)

Training Neural Networks as Recognizers of Formal Languages
by: Butoi, Alexandra, et al.
Published: (2024)

Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns
by: DuSell, Brian, et al.
Published: (2023)

Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling
by: Lipkin, Benjamin, et al.
Published: (2025)

Generalized Measures of Anticipation and Responsivity in Online Language Processing
by: Giulianelli, Mario, et al.
Published: (2024)

Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo
by: Loula, João, et al.
Published: (2025)

Einstein Constants and Smooth Topology
by: LeBrun, Claude
Published: (2025)

Prefix Parsing is Just Parsing
by: Pasti, Clemente, et al.
Published: (2026)

Ensembling Language Models with Sequential Monte Carlo
by: Chan, Robin Shing Moon, et al.
Published: (2026)

Better Estimation of the Kullback--Leibler Divergence Between Language Models
by: Amini, Afra, et al.
Published: (2025)

Syntactic Control of Language Models by Posterior Inference
by: Xefteri, Vicky, et al.
Published: (2025)

PILA: A Historical-Linguistic Dataset of Proto-Italic and Latin
by: Bothwell, Stephen, et al.
Published: (2024)

Desingularizations of Conformally Kaehler, Einstein Orbifolds
by: LeBrun, Claude, et al.
Published: (2026)

Correlation Does Not Imply Compensation: Complexity and Irregularity in the Lexicon
by: Doucette, Amanda, et al.
Published: (2024)

Transducing Language Models
by: Snæbjarnarson, Vésteinn, et al.
Published: (2026)

A Spatio-Temporal Point Process for Fine-Grained Modeling of Reading Behavior
by: Re, Francesco Ignazio, et al.
Published: (2025)

LLM Agents in Interaction: Measuring Personality Consistency and Linguistic Alignment in Interacting Populations of Large Language Models
by: Frisch, Ivar, et al.
Published: (2024)

Projective metric geometry of tropical nuclei: gap matrices, event loci, and order chambers
by: Gastaldi, Juan Luis, et al.
Published: (2026)

Characterizing the Expressivity of Fixed-Precision Transformer Language Models
by: Li, Jiaoda, et al.
Published: (2025)

Transformers Can Represent $n$-gram Language Models
by: Svete, Anej, et al.
Published: (2024)

Gumbel Counterfactual Generation From Language Models
by: Ravfogel, Shauli, et al.
Published: (2024)

Efficiently Computing Susceptibility to Context in Language Models
by: Liu, Tianyu, et al.
Published: (2024)

On the Representational Capacity of Recurrent Neural Language Models
by: Nowak, Franz, et al.
Published: (2023)

A Formal Perspective on Byte-Pair Encoding
by: Zouhar, Vilém, et al.
Published: (2023)

Formal Aspects of Language Modeling
by: Cotterell, Ryan, et al.
Published: (2023)

Exact Hard Monotonic Attention for Character-Level Transduction
by: Wu, Shijie, et al.
Published: (2019)

Cross-lingual, Character-Level Neural Morphological Tagging
by: Cotterell, Ryan, et al.
Published: (2017)

Surprisal Minimisation over Goal-directed Alternatives Predicts Production Choice in Dialogue
by: Utting, Tom, et al.
Published: (2026)

An Algebraic View of the Expressivity of Recurrent Language Models
by: Nowak, Franz, et al.
Published: (2026)

Automating the Analysis of Parsing Algorithms (and other Dynamic Programs)
by: Vieira, Tim, et al.
Published: (2025)

Direct Preference Optimization with an Offset
by: Amini, Afra, et al.
Published: (2024)

Surprise! Uniform Information Density Isn't the Whole Story: Predicting Surprisal Contours in Long-form Discourse
by: Tsipidi, Eleftheria, et al.
Published: (2024)

A Distributional Perspective on Word Learning in Neural Language Models
by: Ficarra, Filippo, et al.
Published: (2025)

Low-Resource Named Entity Recognition with Cross-Lingual, Character-Level Neural Conditional Random Fields
by: Cotterell, Ryan, et al.
Published: (2024)

Bound-state-in-continuum guided modes in a multilayer electro-optically active photonic integrated circuit platform
by: Han, Kyunghun, et al.
Published: (2023)