Imagen de Portada

Reducing Tokenization Premiums for Low-Resource Languages

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Churchill, Geoffrey, Skiena, Steven
Formato:	Preprint
Publicado:	2026
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Evaluating Language Translation Models by Playing Telephone
por: Saba, Syeda Jannatus, et al.
Publicado: (2025)

Hierarchies over Vector Space: Orienting Word and Graph Embeddings
por: Guo, Xingzhi, et al.
Publicado: (2022)

The Shape of Word Embeddings: Quantifying Non-Isometry With Topological Data Analysis
por: Draganov, Ondřej, et al.
Publicado: (2024)

Word Definitions from Large Language Models
por: Pham, Bach, et al.
Publicado: (2023)

Gatsby Without the 'E': Crafting Lipograms with LLMs
por: Balasubramanian, Rohan, et al.
Publicado: (2025)

Stop Jostling: Adaptive Negative Sampling Reduces the Marginalization of Low-Resource Language Tokens by Cross-Entropy Loss
por: Turumtaev, Galim
Publicado: (2026)

Comparative Analysis of Tokenization Algorithms for Low-Resource Language Dzongkha
por: Wangchuk, Tandin, et al.
Publicado: (2025)

Tokenization Strategies for Low-Resource Agglutinative Languages in Word2Vec: Case Study on Turkish and Finnish
por: Hu, Jinfan Frank
Publicado: (2025)

When Every Token Counts: Optimal Segmentation for Low-Resource Language Models
por: Raj, Bharath, et al.
Publicado: (2024)

Trans-Tokenization and Cross-lingual Vocabulary Transfers: Language Adaptation of LLMs for Low-Resource NLP
por: Remy, François, et al.
Publicado: (2024)

Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling
por: Ganescu, Bianca-Mihaela, et al.
Publicado: (2025)

Bootstrapping Embeddings for Low Resource Languages
por: Basoz, Merve, et al.
Publicado: (2026)

The Zeno's Paradox of `Low-Resource' Languages
por: Nigatu, Hellina Hailu, et al.
Publicado: (2024)

GlotLID: Language Identification for Low-Resource Languages
por: Kargaran, Amir Hossein, et al.
Publicado: (2023)

FrugalPrompt: Reducing Contextual Overhead in Large Language Models via Token Attribution
por: Raiyan, Syed Rifat, et al.
Publicado: (2025)

Investigating Hallucination in Conversations for Low Resource Languages
por: Das, Amit, et al.
Publicado: (2025)

On Multilingual Encoder Language Model Compression for Low-Resource Languages
por: Gurgurov, Daniil, et al.
Publicado: (2025)

LLM Probe: Evaluating LLMs for Low-Resource Languages
por: Teklehaymanot, Hailay Kidu, et al.
Publicado: (2026)

Task Arithmetic with Support Languages for Low-Resource ASR
por: Rafkin, Emma, et al.
Publicado: (2026)

LMSpell: Neural Spell Checking for Low-Resource Languages
por: Gunathilake, Akesh, et al.
Publicado: (2025)

Massively Multilingual Text Translation For Low-Resource Languages
por: Zhou, Zhong
Publicado: (2024)

LLMs for Extremely Low-Resource Finno-Ugric Languages
por: Purason, Taido, et al.
Publicado: (2024)

Unsupervised Bilingual Lexicon Induction for Low Resource Languages
por: Rathnayake, Charitha, et al.
Publicado: (2024)

Large Language Models for Toxic Language Detection in Low-Resource Balkan Languages
por: Muminovic, Amel, et al.
Publicado: (2025)

Performance of Recent Large Language Models for a Low-Resourced Language
por: Jayakody, Ravindu, et al.
Publicado: (2024)

Introducing Syllable Tokenization for Low-resource Languages: A Case Study with Swahili
por: Atuhurra, Jesse, et al.
Publicado: (2024)

Utilizing Multilingual Encoders to Improve Large Language Models for Low-Resource Languages
por: Puranegedara, Imalsha, et al.
Publicado: (2025)

Is Small Language Model the Silver Bullet to Low-Resource Languages Machine Translation?
por: Song, Yewei, et al.
Publicado: (2025)

Evaluating and Adapting Large Language Models to Represent Folktales in Low-Resource Languages
por: Meaney, JA, et al.
Publicado: (2024)

Moral Reasoning Across Languages: The Critical Role of Low-Resource Languages in LLMs
por: Zhou, Huichi, et al.
Publicado: (2025)

Cross-Lingual Transfer for Low-Resource Natural Language Processing
por: García-Ferrero, Iker
Publicado: (2025)

Multilingual MFA: Forced Alignment on Low-Resource Related Languages
por: Tosolini, Alessio, et al.
Publicado: (2025)

Exploring Language Model Generalization in Low-Resource Extractive QA
por: Sengupta, Saptarshi, et al.
Publicado: (2024)

Speech-to-Speech Translation Pipelines for Conversations in Low-Resource Languages
por: Popescu-Belis, Andrei, et al.
Publicado: (2025)

Foundation Models for Low-Resource Language Education (Vision Paper)
por: Ding, Zhaojun, et al.
Publicado: (2024)

When Distributions Shifts: Causal Generalization for Low-Resource Languages
por: Aminu, Mahi Aliyu, et al.
Publicado: (2025)

Co-training for Low Resource Scientific Natural Language Inference
por: Sadat, Mobashir, et al.
Publicado: (2024)

Transferring BERT Capabilities from High-Resource to Low-Resource Languages Using Vocabulary Matching
por: Rybak, Piotr
Publicado: (2024)

AI Diffusion in Low Resource Language Countries
por: Misra, Amit, et al.
Publicado: (2025)

LanguaShrink: Reducing Token Overhead with Psycholinguistics
por: Liang, Xuechen, et al.
Publicado: (2024)