Imagen de Portada

BrahmicTokenizer-131K: An Indic-Capable Drop-In Replacement for o200k_base

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autor principal:	Shravan, Rohan
Formato:	Preprint
Publicado:	2026
Materias:	Computation and Language Machine Learning I.2.7
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
por: Shravan, Rohan
Publicado: (2026)

Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction
por: Walker, Nicholas
Publicado: (2024)

Token Erasure as a Footprint of Implicit Vocabulary Items in LLMs
por: Feucht, Sheridan, et al.
Publicado: (2024)

Before the Last Token: Diagnosing Final-Token Safety Probe Failures
por: Doda, Shravan
Publicado: (2026)

Trading Complexity for Expressivity Through Structured Generalized Linear Token Mixing
por: Fagnou, Erwan, et al.
Publicado: (2026)

Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law
por: He, Yanjin, et al.
Publicado: (2025)

Improving the Capabilities of Large Language Model Based Marketing Analytics Copilots With Semantic Search And Fine-Tuning
por: Gao, Yilin, et al.
Publicado: (2024)

Evaluating the Efficacy of Hybrid Deep Learning Models in Distinguishing AI-Generated Text
por: Oketunji, Abiodun Finbarrs
Publicado: (2023)

Improving Influence-based Instruction Tuning Data Selection for Balanced Learning of Diverse Capabilities
por: Dai, Qirun, et al.
Publicado: (2025)

Dialect Matters: Cross-Lingual ASR Transfer for Low-Resource Indic Language Varieties
por: Dhasmana, Akriti, et al.
Publicado: (2026)

TCNN: Triple Convolutional Neural Network Models for Retrieval-based Question Answering System in E-commerce
por: Song, Shuangyong, et al.
Publicado: (2020)

Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design
por: Wrzalik, Marco, et al.
Publicado: (2024)

Ignore Me But Don't Replace Me: Utilizing Non-Linguistic Elements for Pretraining on the Cybersecurity Domain
por: Jang, Eugene, et al.
Publicado: (2024)

ZERA: Zero-init Instruction Evolving Refinement Agent -- From Zero Instructions to Structured Prompts via Principle-based Optimization
por: Yi, Seungyoun, et al.
Publicado: (2025)

Large Language Model (LLM) Bias Index -- LLMBI
por: Oketunji, Abiodun Finbarrs, et al.
Publicado: (2023)

Engineering A Large Language Model From Scratch
por: Oketunji, Abiodun Finbarrs
Publicado: (2024)

Efficient Strategy for Improving Large Language Model (LLM) Capabilities
por: Gutiérrez, Julián Camilo Velandia
Publicado: (2025)

What to Keep and What to Drop: Adaptive Table Filtering Framework
por: Jang, WonJune
Publicado: (2025)

Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding
por: Chen, Haolin, et al.
Publicado: (2024)

Detecting Hallucinations in Large Language Model Generation: A Token Probability Approach
por: Quevedo, Ernesto, et al.
Publicado: (2024)

Beyond Pass@k: Breadth-Depth Metrics for Reasoning Boundaries
por: Dragoi, Marius, et al.
Publicado: (2025)

Towards Latent Diffusion Suitable For Text
por: Midavaine, Nesta, et al.
Publicado: (2026)

SpectralLoRA: Is Low-Frequency Structure Sufficient for LoRA Adaptation? A Spectral Analysis of Weight Updates
por: Singh, Rajveer
Publicado: (2026)

FIM-LoRA: Task-Informative Rank Allocation for LoRA via Calibration-Time Gradient-Variance Estimation
por: Sathyavageeswaran, Ramakrishnan
Publicado: (2026)

Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
por: Borobia, Hector, et al.
Publicado: (2026)

RightNow-Arabic-0.5B-Turbo: An Open Sub-1B Arabic Language Model via Vocabulary Injection and Edge-First Deployment
por: Jaber, Jaber, et al.
Publicado: (2026)

Structured-Sparse Attention for Entity Tracking with Subquadratic Sequence Complexity
por: Zhao, Hangyue, et al.
Publicado: (2026)

PowLU: An Activation Function for Stable Pre-Training of LLMs
por: Jiang, Peijie, et al.
Publicado: (2026)

Resonant Context Anchoring: Decoupling Attention Routing and Signal Gain at Inference Time
por: Zhao, Mingkuan, et al.
Publicado: (2026)

Structure-Guided Entity Resolution: Fine-Tuning LLMs for Robust Name Matching in Complex Linguistic Contexts
por: Chourasia, Shivam, et al.
Publicado: (2026)

RBCorr: Response Bias Correction in Language Models
por: Bhatt, Om, et al.
Publicado: (2026)

Hardware Co-Design Scaling Laws via Roofline Modelling for On-Device LLMs
por: Sun, Luoyang, et al.
Publicado: (2026)

An experimental study of KV cache reuse strategies in chunk-level caching systems
por: Cestola, Samuel, et al.
Publicado: (2026)

Arithmetic OOD Failure Unfolds in Stages in Minimal GPTs
por: Shintani, Seine A.
Publicado: (2026)

Compact Example-Based Explanations for Language Models
por: Schoenegger, Loris, et al.
Publicado: (2026)

Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning
por: Cazares, Manuel Israel
Publicado: (2026)

Enhancing Automated Essay Scoring with Three Techniques: Two-Stage Fine-Tuning, Score Alignment, and Self-Training
por: Choi, Hongseok, et al.
Publicado: (2026)

Decoding-Time Debiasing via Process Reward Models: From Controlled Fill-in to Open-Ended Generation
por: Khan, Muneeb Ur Raheem
Publicado: (2026)

Geometric Deviation as an Unsupervised Pre-Generation Reliability Signal: Probing LLM Representations for Answerability
por: Du, Yucheng
Publicado: (2026)

Better and Worse with Scale: How Contextual Entrainment Diverges with Model Size
por: Kukreja, Dikshant, et al.
Publicado: (2026)