Image de couverture de livre

MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Asgari, Ehsaneddin, Kheir, Yassine El, Javaheri, Mohammad Ali Sadraei
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

SuperPos-Prompt: Enhancing Soft Prompt Tuning of Language Models with Superposition of Multi Token Embeddings
par: SadraeiJavaeri, MohammadAli, et autres
Publié: (2024)

Rethinking Tokenization for Rich Morphology: The Dominance of Unigram over BPE and Morphological Alignment
par: Vemula, Saketh Reddy, et autres
Publié: (2025)

Batching BPE Tokenization Merges
par: Morgan, Alexander P.
Publié: (2024)

CLASP: Contrastive Language-Speech Pretraining for Multilingual Multimodal Information Retrieval
par: Abootorabi, Mohammad Mahdi, et autres
Publié: (2024)

Khayyam Challenge (PersianMMLU): Is Your LLM Truly Wise to The Persian Language?
par: Ghahroodi, Omid, et autres
Publié: (2024)

TuringQ: Benchmarking AI Comprehension in Theory of Computation
par: Zahraei, Pardis Sadat, et autres
Publié: (2024)

EICAP: Deep Dive in Assessment and Enhancement of Large Language Models in Emotional Intelligence through Multi-Turn Conversations
par: Nazar, Nizi, et autres
Publié: (2025)

I Am Aligned, But With Whom? MENA Values Benchmark for Evaluating Cultural Alignment and Multilingual Bias in LLMs
par: Zahraei, Pardis Sadat, et autres
Publié: (2025)

BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training
par: Chizhov, Pavel, et autres
Publié: (2024)

BlockBPE: Parallel BPE Tokenization
par: You, Amos
Publié: (2025)

M$^3$Face: A Unified Multi-Modal Multilingual Framework for Human Face Generation and Editing
par: Mofayezi, Mohammadreza, et autres
Publié: (2024)

GPUTOK: GPU Accelerated Byte Level BPE Tokenization
par: Kadamba, Venu Gopal, et autres
Publié: (2026)

The Judge Who Never Admits: Hidden Shortcuts in LLM-based Evaluation
par: Marioriyad, Arash, et autres
Publié: (2026)

AIMA at SemEval-2024 Task 3: Simple Yet Powerful Emotion Cause Pair Analysis
par: Kure, Alireza Ghahramani, et autres
Publié: (2025)

AIMA at SemEval-2024 Task 10: History-Based Emotion Recognition in Hindi-English Code-Mixed Conversations
par: Abootorabi, Mohammad Mahdi, et autres
Publié: (2025)

MorphNAS: Differentiable Architecture Search for Morphologically-Aware Multilingual NER
par: Devadiga, Prathamesh, et autres
Publié: (2025)

QuranMorph: Morphologically Annotated Quranic Corpus
par: Akra, Diyam, et autres
Publié: (2025)

Constructing a BPE Tokenization DFA
par: Berglund, Martin, et autres
Publié: (2024)

Bridging Linguistic Gaps: Cross-Lingual Mapping in Pre-Training and Dataset for Enhanced Multilingual LLM Performance
par: Zheng, Weihua, et autres
Publié: (2026)

AlphaToken: Decoupling Adaptation and Stability for Path-Aware Response Token Valuation in LLM Post-Training
par: Qing, Liu, et autres
Publié: (2026)

Linguistics-Aware Non-Distortionary LLM Watermarking
par: Park, Shinwoo, et autres
Publié: (2026)

Towards Nepali-language LLMs: Efficient GPT training with a Nepali BPE tokenizer
par: Shrestha, Adarsha, et autres
Publié: (2025)

Ensemble of pre-trained language models and data augmentation for hate speech detection from Arabic tweets
par: Daouadi, Kheir Eddine, et autres
Publié: (2024)

Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
par: Abootorabi, Mohammad Mahdi, et autres
Publié: (2025)

Personality Expression Across Contexts: Linguistic and Behavioral Variation in LLM Agents
par: Han, Bin, et autres
Publié: (2026)

MorphPiece : A Linguistic Tokenizer for Large Language Models
par: Jabbar, Haris
Publié: (2023)

ADAM: A Diverse Archive of Mankind for Evaluating and Enhancing LLMs in Biographical Reasoning
par: Cekinmez, Jasin, et autres
Publié: (2025)

Byte BPE Tokenization as an Inverse string Homomorphism
par: Geng, Saibo, et autres
Publié: (2024)

MorphTok: Morphologically Grounded Tokenization for Indian Languages
par: Brahma, Maharaj, et autres
Publié: (2025)

A Linguistics-Aware LLM Watermarking via Syntactic Predictability
par: Park, Shinwoo, et autres
Publié: (2025)

LiteToken: Removing Intermediate Merge Residues From BPE Tokenizers
par: Sun, Yike, et autres
Publié: (2026)

Language Family Matters: Evaluating LLM-Based ASR Across Linguistic Boundaries
par: Zhang, Yuchen, et autres
Publié: (2026)

Morphological Typology in BPE Subword Productivity and Language Modeling
par: Parra, Iñigo
Publié: (2024)

LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference
par: Fu, Qichen, et autres
Publié: (2024)

Training-Trajectory-Aware Token Selection
par: Shen, Zhanming, et autres
Publié: (2026)

From Characters to Tokens: Dynamic Grouping with Hierarchical BPE
par: Dolga, Rares, et autres
Publié: (2025)

AdaptBPE: From General Purpose to Specialized Tokenizers
par: Liyanage, Vijini, et autres
Publié: (2026)

Training Text-to-Molecule Models with Context-Aware Tokenization
par: Kim, Seojin, et autres
Publié: (2025)

Token-Budget-Aware LLM Reasoning
par: Han, Tingxu, et autres
Publié: (2024)

HALF: Harm-Aware LLM Fairness Evaluation Aligned with Deployment
par: Mekky, Ali, et autres
Publié: (2025)