Cover Image

MorphPiece : A Linguistic Tokenizer for Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Author:	Jabbar, Haris
Format:	Preprint
Published:	2023
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MorphTok: Morphologically Grounded Tokenization for Indian Languages
by: Brahma, Maharaj, et al.
Published: (2025)

Towards Linguistically-Aware and Language-Independent Tokenization for Large Language Models (LLMs)
by: Rahman, Abrar, et al.
Published: (2024)

MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies
by: Asgari, Ehsaneddin, et al.
Published: (2025)

Piecing Together Clues: A Benchmark for Evaluating the Detective Skills of Large Language Models
by: Gu, Zhouhong, et al.
Published: (2023)

Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Efforts
by: Tan, Zhaoxuan, et al.
Published: (2024)

A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications
by: Guan, Jian, et al.
Published: (2025)

Linguistically Informed Tokenization Improves ASR for Underresourced Languages
by: Daul, Massimo, et al.
Published: (2025)

DetectBench: Can Large Language Model Detect and Piece Together Implicit Evidence?
by: Gu, Zhouhong, et al.
Published: (2024)

Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models
by: Zhou, Xinyu, et al.
Published: (2024)

Which Pieces Does Unigram Tokenization Really Need?
by: Land, Sander, et al.
Published: (2025)

How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers
by: Roberts, Jonathan, et al.
Published: (2026)

Linguistic Intelligence in Large Language Models for Telecommunications
by: Ahmed, Tasnim, et al.
Published: (2024)

Unveiling Linguistic Regions in Large Language Models
by: Zhang, Zhihao, et al.
Published: (2024)

Benchmarking Linguistic Diversity of Large Language Models
by: Guo, Yanzhu, et al.
Published: (2024)

Problematic Tokens: Tokenizer Bias in Large Language Models
by: Yang, Jin, et al.
Published: (2024)

Multi-Objective Linguistic Control of Large Language Models
by: Nguyen, Dang, et al.
Published: (2024)

Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models
by: Yang, Jinbiao
Published: (2024)

Inductive Linguistic Reasoning with Large Language Models
by: Ramji, Raghav, et al.
Published: (2024)

The Shrinking Landscape of Linguistic Diversity in the Age of Large Language Models
by: Sourati, Zhivar, et al.
Published: (2025)

Large Language Models as Proxies for Theories of Human Linguistic Cognition
by: Ziv, Imry, et al.
Published: (2025)

Brain in a Vat: On Missing Pieces Towards Artificial General Intelligence in Large Language Models
by: Ma, Yuxi, et al.
Published: (2023)

Large Language Model as Token Compressor and Decompressor
by: Li, Wenbing, et al.
Published: (2026)

Retrofitting Large Language Models with Dynamic Tokenization
by: Feher, Darius, et al.
Published: (2024)

A Comparative Approach to Assessing Linguistic Creativity of Large Language Models and Humans
by: Dinu, Anca, et al.
Published: (2025)

Performance Evaluation of Tokenizers in Large Language Models for the Assamese Language
by: Tamang, Sagar, et al.
Published: (2024)

Generative Linguistics, Large Language Models, and the Social Nature of Scientific Success
by: Hao, Sophie
Published: (2025)

Probing Large Language Models in Reasoning and Translating Complex Linguistic Puzzles
by: Lin, Zheng-Lin, et al.
Published: (2025)

Are Large Language Models the future crowd workers of Linguistics?
by: Ferrazzo, Iris
Published: (2025)

MorphNLI: A Stepwise Approach to Natural Language Inference Using Text Morphing
by: Negru, Vlad Andrei, et al.
Published: (2025)

Rethinking Personalization in Large Language Models at the Token Level
by: Zhang, Chenheng, et al.
Published: (2026)

Linguistic Blind Spots of Large Language Models
by: Cheng, Jiali, et al.
Published: (2025)

FTP: A Fine-grained Token-wise Pruner for Large Language Models via Token Routing
by: Li, Zekai, et al.
Published: (2024)

Unified Large Language Models for Misinformation Detection in Low-Resource Linguistic Settings
by: Islam, Muhammad, et al.
Published: (2025)

Decomposed Prompting: Probing Multilingual Linguistic Structure Knowledge in Large Language Models
by: Nie, Ercong, et al.
Published: (2024)

The Invalsi Benchmarks: measuring Linguistic and Mathematical understanding of Large Language Models in Italian
by: Puccetti, Giovanni, et al.
Published: (2024)

Investigating Large Language Models' Linguistic Abilities for Text Preprocessing
by: Braga, Marco, et al.
Published: (2025)

Detecting Linguistic Indicators for Stereotype Assessment with Large Language Models
by: Görge, Rebekka, et al.
Published: (2025)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
by: Wang, Dixuan, et al.
Published: (2024)

UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu
by: Adeeba, Farah, et al.
Published: (2025)

A Methodology for Explainable Large Language Models with Integrated Gradients and Linguistic Analysis in Text Classification
by: Ribeiro, Marina, et al.
Published: (2024)