Cover Image

Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Dang, Thao Anh, Raviv, Limor, Galke, Lukas
Format:	Preprint
Published:	2024
Subjects:	Computation and Language I.2.7
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Learning and communication pressures in neural networks: Lessons from emergent communication
by: Galke, Lukas, et al.
Published: (2024)

What makes a language easy to deep-learn? Deep neural networks and humans similarly benefit from compositional structure
by: Galke, Lukas, et al.
Published: (2023)

Efficacy of ByT5 in Multilingual Translation of Biblical Texts for Underrepresented Languages
by: Aars, Corinne, et al.
Published: (2024)

idT5: Indonesian Version of Multilingual T5 Transformer
by: Fuadi, Mukhlish, et al.
Published: (2023)

Training Language Models to Use Prolog as a Tool
by: Mellgren, Niklas, et al.
Published: (2025)

Not Everything That Counts Can Be Counted: A Case for Safe Qualitative AI
by: Beltoft, Stine, et al.
Published: (2025)

What Drives Performance in Multilingual Language Models?
by: Nezhad, Sina Bagheri, et al.
Published: (2024)

Socially Responsible Data for Large Multilingual Language Models
by: Smart, Andrew, et al.
Published: (2024)

Morphological Analysis for the Maltese Language: The Challenges of a Hybrid System
by: Borg, Claudia, et al.
Published: (2017)

RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs
by: Saji, Alan, et al.
Published: (2025)

Dialect Normalization using Large Language Models and Morphological Rules
by: Dimakis, Antonios, et al.
Published: (2025)

Ensemble Language Models for Multilingual Sentiment Analysis
by: Hasan, Md Arid
Published: (2024)

Exploring the Maze of Multilingual Modeling
by: Nezhad, Sina Bagheri, et al.
Published: (2023)

Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspect
by: Klerings, Alina, et al.
Published: (2025)

DimStance: Multilingual Datasets for Dimensional Stance Analysis
by: Becker, Jonas, et al.
Published: (2026)

Ensembling Multilingual Transformers for Robust Sentiment Analysis of Tweets
by: Bilehsavar, Meysam Shirdel, et al.
Published: (2025)

LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models
by: Pitorro, Hugo, et al.
Published: (2025)

Sigmoid Head for Quality Estimation under Language Ambiguity
by: Dinh, Tu Anh, et al.
Published: (2026)

Credit C-GPT: A Domain-Specialized Large Language Model for Conversational Understanding in Vietnamese Debt Collection
by: Hong, Nhung Nguyen Thi, et al.
Published: (2026)

Are Generative Models Underconfident? Better Quality Estimation with Boosted Model Probability
by: Dinh, Tu Anh, et al.
Published: (2025)

Adapting Multilingual Models to Code-Mixed Tasks via Model Merging
by: Kodali, Prashant, et al.
Published: (2025)

2M-BELEBELE: Highly Multilingual Speech and American Sign Language Comprehension Dataset
by: Costa-jussà, Marta R., et al.
Published: (2024)

DimABSA: Building Multilingual and Multidomain Datasets for Dimensional Aspect-Based Sentiment Analysis
by: Lee, Lung-Hao, et al.
Published: (2026)

The Effect of Data Partitioning Strategy on Model Generalizability: A Case Study of Morphological Segmentation
by: Liu, Zoey, et al.
Published: (2024)

Comparative Study of Large Language Models on Chinese Film Script Continuation: An Empirical Analysis Based on GPT-5.2 and Qwen-Max
by: Cao, Yuxuan, et al.
Published: (2026)

Towards Massive Multilingual Holistic Bias
by: Tan, Xiaoqing Ellen, et al.
Published: (2024)

Towards Fundamental Language Models: Does Linguistic Competence Scale with Model Size?
by: Collado-Montañez, Jaime, et al.
Published: (2025)

ML-Promise: A Multilingual Dataset for Corporate Promise Verification
by: Seki, Yohei, et al.
Published: (2024)

Quality Estimation with $k$-nearest Neighbors and Automatic Evaluation for Model-specific Quality Estimation
by: Dinh, Tu Anh, et al.
Published: (2024)

CRISP: Persistent Concept Unlearning via Sparse Autoencoders
by: Ashuach, Tomer, et al.
Published: (2025)

All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens
by: Mamidanna, Siddarth, et al.
Published: (2025)

Personality, Role, and Expressive Style in Large Language Models: An Interactionist Analysis
by: Nagao, Moe, et al.
Published: (2026)

Predictive Simultaneous Interpretation: Harnessing Large Language Models for Democratizing Real-Time Multilingual Communication
by: Iida, Kurando, et al.
Published: (2024)

Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks
by: Nielsen, Dan Saattrup, et al.
Published: (2024)

Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction
by: Walker, Nicholas
Published: (2024)

Egalitarian Language Representation in Language Models: It All Begins with Tokenizers
by: Velayuthan, Menan, et al.
Published: (2024)

The Provenance Gap in Clinical AI: Evidence-Traceable Temporal Knowledge Graphs for Rare Disease Reasoning
by: Ahmed, Md Shamim, et al.
Published: (2026)

I run as fast as a rabbit, can you? A Multilingual Simile Dialogue Dataset
by: Ma, Longxuan, et al.
Published: (2023)

Comonadic Morphophonology: A Compositional Framework for Context-Dependent Morphological Rules in Finnish
by: Jang, Yongseok
Published: (2026)

Comparing Complex Concepts with Transformers: Matching Patent Claims Against Natural Language Text
by: Blume, Matthias, et al.
Published: (2024)