Copertina

Evaluating Morphological Plausibility of Subword Tokenization via Statistical Alignment with Morpho-Syntactic Features

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Stephen, Abishek, Libovický, Jindřich
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Lexically Grounded Subword Segmentation
di: Libovický, Jindřich, et al.
Pubblicazione: (2024)

On the Credibility of Evaluating LLMs using Survey Questions
di: Libovický, Jindřich
Pubblicazione: (2026)

Understanding Cross-Lingual Alignment -- A Survey
di: Hämmerl, Katharina, et al.
Pubblicazione: (2024)

Beyond Literal Token Overlap: Token Alignability for Multilinguality
di: Hämmerl, Katharina, et al.
Pubblicazione: (2025)

Crowdsourcing Piedmontese to Test LLMs on Non-Standard Orthography
di: Vico, Gianluca, et al.
Pubblicazione: (2026)

LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
di: Eigler, Lukáš, et al.
Pubblicazione: (2026)

Different Time, Different Language: Revisiting the Bias Against Non-Native Speakers in GPT Detectors
di: Ali, Adnan Al, et al.
Pubblicazione: (2026)

Multilingual Vision-Language Models, A Survey
di: Manea, Andrei-Alexandru, et al.
Pubblicazione: (2025)

Investigating the Effect of Parallel Data in the Cross-Lingual Transfer for Vision-Language Encoders
di: Manea, Andrei-Alexandru, et al.
Pubblicazione: (2025)

How Gender Interacts with Political Values: A Case Study on Czech BERT Models
di: Ali, Adnan Al, et al.
Pubblicazione: (2024)

CUS-QA: Local-Knowledge-Oriented Open-Ended Question Answering Dataset
di: Libovický, Jindřich, et al.
Pubblicazione: (2025)

Token Alignment via Character Matching for Subword Completion
di: Athiwaratkun, Ben, et al.
Pubblicazione: (2024)

Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge
di: Batsuren, Khuyagbaatar, et al.
Pubblicazione: (2024)

Conditional Unigram Tokenization with Parallel Data
di: Vico, Gianluca, et al.
Pubblicazione: (2025)

Distantly Supervised Morpho-Syntactic Model for Relation Extraction
di: Gutehrlé, Nicolas, et al.
Pubblicazione: (2024)

A Joint Multitask Model for Morpho-Syntactic Parsing
di: Inostroza, Demian, et al.
Pubblicazione: (2025)

Teaching LLMs at Charles University: Assignments and Activities
di: Helcl, Jindřich, et al.
Pubblicazione: (2024)

Evaluating Morphological Alignment of Tokenizers in 70 Languages
di: Arnett, Catherine, et al.
Pubblicazione: (2025)

Subword Tokenization Strategies for Kurdish Word Embeddings
di: Salehi, Ali, et al.
Pubblicazione: (2025)

Evaluating Subword Tokenization Techniques for Bengali: A Benchmark Study with BengaliBPE
di: Patwary, Firoj Ahmmed, et al.
Pubblicazione: (2025)

The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages
di: Meyer, Francois, et al.
Pubblicazione: (2025)

Morphological Typology in BPE Subword Productivity and Language Modeling
di: Parra, Iñigo
Pubblicazione: (2024)

Enhancing Conceptual Understanding in Multimodal Contrastive Learning through Hard Negative Samples
di: Rösch, Philipp J., et al.
Pubblicazione: (2024)

Tokenization Falling Short: On Subword Robustness in Large Language Models
di: Chai, Yekun, et al.
Pubblicazione: (2024)

Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
di: Gigant, Théo, et al.
Pubblicazione: (2026)

Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay
di: Altinok, Duygu
Pubblicazione: (2026)

Targeted Syntactic Evaluation of Language Models on Georgian Case Alignment
di: Gallagher, Daniel, et al.
Pubblicazione: (2026)

Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT
di: Wolleb, Benoist, et al.
Pubblicazione: (2023)

Charles Translator: A Machine Translation System between Ukrainian and Czech
di: Popel, Martin, et al.
Pubblicazione: (2024)

SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers
di: Chelombitko, Iaroslav, et al.
Pubblicazione: (2026)

Team Ryu's Submission to SIGMORPHON 2024 Shared Task on Subword Tokenization
di: Li, Zilong
Pubblicazione: (2024)

Multilingual Text-to-Image Generation Magnifies Gender Stereotypes and Prompt Engineering May Not Help You
di: Friedrich, Felix, et al.
Pubblicazione: (2024)

MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
di: Asgari, Ehsaneddin, et al.
Pubblicazione: (2025)

Automated Quality Control for Language Documentation: Detecting Phonotactic Inconsistencies in a Kokborok Wordlist
di: van Dam, Kellen Parker, et al.
Pubblicazione: (2025)

Defense Against Syntactic Textual Backdoor Attacks with Token Substitution
di: Li, Xinglin, et al.
Pubblicazione: (2024)

T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings
di: Deiseroth, Björn, et al.
Pubblicazione: (2024)

Distributional Properties of Subword Regularization
di: Cognetta, Marco, et al.
Pubblicazione: (2024)

Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
di: Vemula, Saketh Reddy, et al.
Pubblicazione: (2025)

Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods
di: Suyunu, Burak, et al.
Pubblicazione: (2024)

A Computational Operationalisation of Competing Maturational Theories of Syntactic Development via Statistical Grammar Induction
di: Marcheva, Mila, et al.
Pubblicazione: (2026)