Image de couverture de livre

Tokenization as Finite-State Transduction

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Cognetta, Marco, Okazaki, Naoaki
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Formal Languages and Automata Theory
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Tutorial: $φ$-Transductions in OpenFst via the Gallic Semiring
par: Cognetta, Marco, et autres
Publié: (2025)

Comparing Spoken Languages using Paninian System of Sounds and Finite State Machines
par: Prabhu, Shreekanth M, et autres
Publié: (2023)

Fast and General Automatic Differentiation for Finite-State Methods
par: Yang, Lucas Ondel, et autres
Publié: (2026)

Efficient Constructions of Finite-State Independent Normal Pairs
par: Pulari, Subin
Publié: (2026)

Active Inference of Extended Finite State Machine Models with Registers and Guards
par: Groz, Roland, et autres
Publié: (2026)

The Quadratic State Cost of Classical Simulation of One-Way Quantum Finite Automata
par: Chen, Zeyu, et autres
Publié: (2026)

Studying homing and synchronizing sequences for Timed Finite State Machines with output delays
par: Vinarskii, Evgenii, et autres
Publié: (2025)

Approximate Problems for Finite Transducers
par: Filiot, Emmanuel, et autres
Publié: (2025)

Token Games and History-Deterministic Quantitative-Automata
par: Boker, Udi, et autres
Publié: (2021)

Autoformalization in the Wild: Assessing LLMs on Real-World Mathematical Definitions
par: Zhang, Lan, et autres
Publié: (2025)

MASA: LLM-Driven Multi-Agent Systems for Autoformalization
par: Zhang, Lan, et autres
Publié: (2025)

Finite Automata for Efficient Graph Recognition
par: Drewes, Frank, et autres
Publié: (2024)

Finite-valued Streaming String Transducers
par: Filiot, Emmanuel, et autres
Publié: (2024)

Repetitive Finite Automata With Translucent Letters
par: Mráz, František, et autres
Publié: (2024)

Finite maximal codes and factorizations of cyclic groups
par: De Felice, Clelia
Publié: (2022)

Descriptional Complexity of Finite Automata -- Selected Highlights
par: Salomaa, Arto, et autres
Publié: (2023)

Multihead Finite-State Compression
par: Lutz, Neil
Publié: (2025)

Measure Many Quantum Finite Automata on Infinite Words
par: Midya, Abhisek, et autres
Publié: (2026)

The 2-Token Theorem: Recognising History-Deterministic Parity Automata Efficiently
par: Lehtinen, Karoliina, et autres
Publié: (2025)

Certified Symbolic Finite Transducers: Formalization and Applications to String Analysis
par: Kan, Shuanglong, et autres
Publié: (2025)

Optimal Approximate Minimization of One-Letter Weighted Finite Automata
par: Lacroce, Clara, et autres
Publié: (2023)

Reversible Weighted Automata over Finite Rings and Monoids with Commuting Idempotents
par: Kostolányi, Peter, et autres
Publié: (2026)

Mata, a Fast and Simple Finite Automata Library (Technical Report)
par: Chocholatý, David, et autres
Publié: (2023)

Synthesizing Petri Nets from Labelled Petri Nets using Token Trail Regions
par: Bergenthum, Robin, et autres
Publié: (2026)

Distributional Learning of Context-Free Languages under Fixed Finite-Monoid Typing
par: Kuriyama, Takayuki
Publié: (2014)

Finite-State Automaton To/From Regular Expression Visualization
par: Morazán, Marco T., et autres
Publié: (2024)

Generalised Arc Consistency via the Synchronised Product of Finite Automata wrt a Constraint
par: Beldiceanu, Nicolas
Publié: (2025)

Finite Automata Encoding Piecewise Polynomials
par: Berdinsky, Dmitry, et autres
Publié: (2021)

GrappaRE -- A Tool for Efficient Graph Recognition Based on Finite Automata and Regular Expressions
par: De Rosa, Mattia, et autres
Publié: (2025)

Complexity of Unary Exclusive Nondeterministic Finite Automata
par: Kutrib, Martin, et autres
Publié: (2024)

Power of Counting by Nonuniform Families of Polynomial-Size Finite Automata
par: Yamakami, Tomoyuki
Publié: (2023)

Constructing a BPE Tokenization DFA
par: Berglund, Martin, et autres
Publié: (2024)

Operational State Complexity of Block Languages
par: Duarte, Guilherme, et autres
Publié: (2024)

On the Representation and State Complexity of Block Languages
par: Duarte, Guilherme, et autres
Publié: (2024)

Waiting Nets: State Classes and Taxonomy
par: Hélouët, Loïc, et autres
Publié: (2022)

Exact Accepting-State Spectrum for Reversal of Permutation Automata
par: German, Samuel
Publié: (2026)

Languages of Boundedly-Ambiguous Vector Addition Systems with States
par: Czerwiński, Wojciech, et autres
Publié: (2025)

Unambiguous and Co-Nondeterministic Computations of Finite Automata and Pushdown Automata Families and the Effects of Multiple Counters
par: Yamakami, Tomoyuki
Publié: (2024)

State estimation of timed automata under partial observation [Draft version]
par: Gao, Chao, et autres
Publié: (2024)

Verification and Enforcement of Strong State-Based Opacity for Discrete-Event Systems
par: Han, Xiaoguang, et autres
Publié: (2024)