Tutorial: $φ$-Transductions in OpenFst via the Gallic Semiring
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909656224366592 |
|---|---|
| author | Cognetta, Marco Allauzen, Cyril |
| author_facet | Cognetta, Marco Allauzen, Cyril |
| contents | OpenFst, a popular finite-state transducer library, supports $φ$-transitions but, due to an implementation constraint, they cannot be used with transducers in a straightforward way.
In this short tutorial, we describe how one can use other functionality provided by OpenFst (namely, the Gallic semiring) to correctly implement $φ$-transductions and demonstrate it by implementing the MaxMatch (WordPiece) tokenization algorithm (Devlin et al., 2019; Song et al., 2021). Accompanying self-contained code examples are provided. https://www.openfst.org/twiki/pub/Contrib/FstContrib/phi_transduction_tutorial_code.tgz |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_17942 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Tutorial: $φ$-Transductions in OpenFst via the Gallic Semiring Cognetta, Marco Allauzen, Cyril Formal Languages and Automata Theory Computation and Language OpenFst, a popular finite-state transducer library, supports $φ$-transitions but, due to an implementation constraint, they cannot be used with transducers in a straightforward way. In this short tutorial, we describe how one can use other functionality provided by OpenFst (namely, the Gallic semiring) to correctly implement $φ$-transductions and demonstrate it by implementing the MaxMatch (WordPiece) tokenization algorithm (Devlin et al., 2019; Song et al., 2021). Accompanying self-contained code examples are provided. https://www.openfst.org/twiki/pub/Contrib/FstContrib/phi_transduction_tutorial_code.tgz |
| title | Tutorial: $φ$-Transductions in OpenFst via the Gallic Semiring |
| topic | Formal Languages and Automata Theory Computation and Language |
| url | https://arxiv.org/abs/2506.17942 |