Cover Image

BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base

Fuente: arXiv

Saved in:

Bibliographic Details
Main Author:	Shravan, Rohan
Format:	Preprint
Published:	2026
Subjects:	Computation and Language Machine Learning I.2.7
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
by: Shravan, Rohan
Published: (2026)

Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction
by: Walker, Nicholas
Published: (2024)

Before the Last Token: Diagnosing Final-Token Safety Probe Failures
by: Doda, Shravan
Published: (2026)

Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs
by: Feucht, Sheridan, et al.
Published: (2024)

Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
by: Fagnou, Erwan, et al.
Published: (2026)

Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law
by: He, Yanjin, et al.
Published: (2025)

Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning
by: Gao, Yilin, et al.
Published: (2024)

Evaluating the Efficacy of Hybrid Deep Learning Models in Distinguishing AI-Generated Text
by: Oketunji, Abiodun Finbarrs
Published: (2023)

Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities
by: Dai, Qirun, et al.
Published: (2025)

Dialect Matters: Cross-Lingual ASR Transfer for Low-Resource Indic Language Varieties
by: Dhasmana, Akriti, et al.
Published: (2026)

TCNN: Triple Convolutional Neural Network Models for Retrieval-based Question Answering System in E-commerce
by: Song, Shuangyong, et al.
Published: (2020)

Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design
by: Wrzalik, Marco, et al.
Published: (2024)

Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain
by: Jang, Eugene, et al.
Published: (2024)

ZERA: Zero-init Instruction Evolving Refinement Agent -- From Zero Instructions to Structured Prompts via Principle-based Optimization
by: Yi, Seungyoun, et al.
Published: (2025)

Large Language Model (LLM) Bias Index -- LLMBI
by: Oketunji, Abiodun Finbarrs, et al.
Published: (2023)

Engineering A Large Language Model From Scratch
by: Oketunji, Abiodun Finbarrs
Published: (2024)

Efficient Strategy for Improving Large Language Model (LLM) Capabilities
by: Gutiérrez, Julián Camilo Velandia
Published: (2025)

What to Keep and What to Drop: Adaptive Table Filtering Framework
by: Jang, WonJune
Published: (2025)

Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding
by: Chen, Haolin, et al.
Published: (2024)

Detecting Hallucinations in Large Language Model Generation: A Token Probability Approach
by: Quevedo, Ernesto, et al.
Published: (2024)

Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries
by: Dragoi, Marius, et al.
Published: (2025)

Towards Latent Diffusion Suitable For Text
by: Midavaine, Nesta, et al.
Published: (2026)

SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates
by: Singh, Rajveer
Published: (2026)

FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation
by: Sathyavageeswaran, Ramakrishnan
Published: (2026)

Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
by: Borobia, Hector, et al.
Published: (2026)

RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment
by: Jaber, Jaber, et al.
Published: (2026)

Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
by: Zhao, Hangyue, et al.
Published: (2026)

PowLU: An Activation Function for Stable Pre-Training of LLMs
by: Jiang, Peijie, et al.
Published: (2026)

Resonant Context Anchoring: Decoupling Attention Routing and Signal Gain at Inference Time
by: Zhao, Mingkuan, et al.
Published: (2026)

Structure-Guided Entity Resolution: Fine-Tuning LLMs for Robust Name Matching in Complex Linguistic Contexts
by: Chourasia, Shivam, et al.
Published: (2026)

RBCorr: Response Bias Correction in Language Models
by: Bhatt, Om, et al.
Published: (2026)

Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMs
by: Sun, Luoyang, et al.
Published: (2026)

An experimental study of KV cache reuse strategies in chunk-level caching systems
by: Cestola, Samuel, et al.
Published: (2026)

Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs
by: Shintani, Seine A.
Published: (2026)

Compact Example-Based Explanations for Language Models
by: Schoenegger, Loris, et al.
Published: (2026)

Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning
by: Cazares, Manuel Israel
Published: (2026)

Enhancing Automated Essay Scoring with Three Techniques: Two-Stage Fine-Tuning, Score Alignment, and Self-Training
by: Choi, Hongseok, et al.
Published: (2026)

Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation
by: Khan, Muneeb Ur Raheem
Published: (2026)

Geometric Deviation as an Unsupervised Pre-Generation Reliability Signal: Probing LLM Representations for Answerability
by: Du, Yucheng
Published: (2026)

Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size
by: Kukreja, Dikshant, et al.
Published: (2026)