Cover Image

When Every Token Counts: Optimal Segmentation for Low-Resource Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Raj, Bharath, Suri, Garvit, Dewangan, Vikrant, Sonavane, Raghav
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

DENIAHL: In-Context Features Influence LLM Needle-In-A-Haystack Abilities
by: Dai, Hui, et al.
Published: (2024)

Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling
by: Ganescu, Bianca-Mihaela, et al.
Published: (2025)

TIDE: Every Layer Knows the Token Beneath the Context
by: Jaiswal, Ajay, et al.
Published: (2026)

Kakugo: Distillation of Low-Resource Languages into Small Language Models
by: Devine, Peter, et al.
Published: (2026)

When Can Transformers Count to n?
by: Yehudai, Gilad, et al.
Published: (2024)

Every Response Counts: Quantifying Uncertainty of LLM-based Multi-Agent Systems through Tensor Decomposition
by: Chen, Tiejin, et al.
Published: (2026)

Every Character Counts: From Vulnerability to Defense in Phishing Detection
by: Chiper, Maria, et al.
Published: (2025)

Are More Tokens Rational? Inference-Time Scaling in Language Models as Adaptive Resource Rationality
by: Hu, Zhimin, et al.
Published: (2026)

Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs
by: Ling Team, et al.
Published: (2025)

Large Multimodal Models for Low-Resource Languages: A Survey
by: Lupascu, Marian, et al.
Published: (2025)

Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
by: Kim, Gyeongman, et al.
Published: (2025)

RLHF: A comprehensive Survey for Cultural, Multimodal and Low Latency Alignment Methods
by: Sharma, Raghav, et al.
Published: (2025)

Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts
by: Somayajula, Sai Ashish, et al.
Published: (2024)

Contrastive Decoding for Synthetic Data Generation in Low-Resource Language Modeling
by: Ulm, Jannek, et al.
Published: (2025)

Small Language Models for Privacy-Preserving Clinical Information Extraction in Low-Resource Languages
by: Ghaffarzadeh-Esfahani, Mohammadreza, et al.
Published: (2026)

Emotion Classification in Low and Moderate Resource Languages
by: Tafreshi, Shabnam, et al.
Published: (2024)

AfroXLMR-Comet: Multilingual Knowledge Distillation with Attention Matching for Low-Resource languages
by: Raju, Joshua Sakthivel, et al.
Published: (2025)

Few-Shot Cross-Lingual Transfer for Prompting Large Language Models in Low-Resource Languages
by: Toukmaji, Christopher
Published: (2024)

Length-MAX Tokenizer for Language Models
by: Dong, Dong, et al.
Published: (2025)

Make Every Draft Count: Hidden State based Speculative Decoding
by: Chen, Yuetao, et al.
Published: (2026)

Dependency Parsing Across the Resource Spectrum: Evaluating Architectures on High and Low-Resource Languages
by: Guan, Kevin, et al.
Published: (2026)

RoboSignature: Robust Signature and Watermarking on Network Attacks
by: Shaan, Aryaman, et al.
Published: (2024)

Understanding and Mitigating Tokenization Bias in Language Models
by: Phan, Buu, et al.
Published: (2024)

Adapting Language Models via Token Translation
by: Feng, Zhili, et al.
Published: (2024)

Counterfactual Token Generation in Large Language Models
by: Chatzi, Ivi, et al.
Published: (2024)

Guiding Language Model Reasoning with Planning Tokens
by: Wang, Xinyi, et al.
Published: (2023)

A Multilingual Sentiment Lexicon for Low-Resource Language Translation using Large Languages Models and Explainable AI
by: Malinga, Melusi, et al.
Published: (2024)

LexC-Gen: Generating Data for Extremely Low-Resource Languages with Large Language Models and Bilingual Lexicons
by: Yong, Zheng-Xin, et al.
Published: (2024)

Bridging the Bosphorus: Advancing Turkish Large Language Models through Strategies for Low-Resource Language Adaptation and Benchmarking
by: Acikgoz, Emre Can, et al.
Published: (2024)

Scaling Optimal LR Across Token Horizons
by: Bjorck, Johan, et al.
Published: (2024)

Investigating the Transferability of Code Repair for Low-Resource Programming Languages
by: Wong, Kyle, et al.
Published: (2024)

Every Token Counts: Generalizing 16M Ultra-Long Context in Large Language Models
by: Hu, Xiang, et al.
Published: (2025)

Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scaling
by: Wang, Xinglin, et al.
Published: (2025)

TharuChat: Bootstrapping Large Language Models for a Low-Resource Language via Synthetic Data and Human Validation
by: Panth, Prajwal, et al.
Published: (2026)

Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding
by: Zhang, Zhongjian, et al.
Published: (2026)

Token-Efficient Leverage Learning in Large Language Models
by: Zeng, Yuanhao, et al.
Published: (2024)

Language Model Cascades: Token-level uncertainty and beyond
by: Gupta, Neha, et al.
Published: (2024)

Compositional Steering of Large Language Models with Steering Tokens
by: Radevski, Gorjan, et al.
Published: (2026)

Probabilistic Token Alignment for Large Language Model Fusion
by: Zeng, Runjia, et al.
Published: (2025)

Specialised or Generic? Tokenization Choices for Radiology Language Models
by: Warr, Hermione, et al.
Published: (2025)