Image de couverture de livre

AraToken: Optimizing Arabic Tokenization with Normalization Pipeline and Language Extension for Qwen3

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Kashirskiy, Mark, Lipinski, Artiom, Makarov, Ilya
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Artificial Intelligence I.2.7
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Egalitarian Language Representation in Language Models: It All Begins with Tokenizers
par: Velayuthan, Menan, et autres
Publié: (2024)

Advancing Polish Language Modeling through Tokenizer Optimization in the Bielik v3 7B and 11B Series
par: Ociepa, Krzysztof, et autres
Publié: (2026)

Beyond Token Length: Step Pruner for Efficient and Accurate Reasoning in Large Language Models
par: Wu, Canhui, et autres
Publié: (2025)

SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM
par: Wang, Quandong, et autres
Publié: (2024)

Next Token Prediction Is a Dead End for Creativity
par: Olatunji, Ibukun, et autres
Publié: (2025)

QuickSilver -- Speeding up LLM Inference through Dynamic Token Halting, KV Skipping, Contextual Token Fusion, and Adaptive Matryoshka Quantization
par: Khanna, Danush, et autres
Publié: (2025)

Tokenization Is More Than Compression
par: Schmidt, Craig W., et autres
Publié: (2024)

Cultural Benchmarking of LLMs in Standard and Dialectal Arabic Dialogues
par: Kautsar, Muhammad Dehan Al, et autres
Publié: (2026)

Detecting Hallucinations in Large Language Model Generation: A Token Probability Approach
par: Quevedo, Ernesto, et autres
Publié: (2024)

Graphemic Normalization of the Perso-Arabic Script
par: Doctor, Raiomond, et autres
Publié: (2022)

The Good, the Bad, and the Hulk-like GPT: Analyzing Emotional Decisions of Large Language Models in Cooperation and Bargaining Games
par: Mozikov, Mikhail, et autres
Publié: (2024)

SuS: Strategy-aware Surprise for Intrinsic Exploration
par: Kashirskiy, Mark, et autres
Publié: (2026)

TREX: Tokenizer Regression for Optimal Data Mixture
par: Won, Inho, et autres
Publié: (2026)

Why Models Know But Don't Say: Chain-of-Thought Faithfulness Divergence Between Thinking Tokens and Answers in Open-Weight Reasoning Models
par: Young, Richard J.
Publié: (2026)

RomanLens: The Role Of Latent Romanization In Multilinguality In LLMs
par: Saji, Alan, et autres
Publié: (2025)

Text-Based Approaches to Item Difficulty Modeling in Large-Scale Assessments: A Systematic Review
par: Peters, Sydney, et autres
Publié: (2025)

Qtok: A Comprehensive Framework for Evaluating Multilingual Tokenizer Quality in Large Language Models
par: Chelombitko, Iaroslav, et autres
Publié: (2024)

LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Models
par: Pitorro, Hugo, et autres
Publié: (2025)

SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmark
par: Dahir, Khalid Yusuf
Publié: (2026)

Attentive Reasoning Queries: A Systematic Method for Optimizing Instruction-Following in Large Language Models
par: Karov, Bar, et autres
Publié: (2025)

Bielik 11B v3: Multilingual Large Language Model for European Languages
par: Ociepa, Krzysztof, et autres
Publié: (2025)

Control Reinforcement Learning: Interpretable Token-Level Steering of LLMs via Sparse Autoencoder Features
par: Cho, Seonglae, et autres
Publié: (2026)

Search-R3: Unifying Reasoning and Embedding in Large Language Models
par: Gui, Yuntao, et autres
Publié: (2025)

Word Importance Explains How Prompts Affect Language Model Outputs
par: Hackmann, Stefan, et autres
Publié: (2024)

Large Language Model (LLM) Bias Index -- LLMBI
par: Oketunji, Abiodun Finbarrs, et autres
Publié: (2023)

AraDiCE: Benchmarks for Dialectal and Cultural Capabilities in LLMs
par: Mousi, Basel, et autres
Publié: (2024)

All for One: LLMs Solve Mental Math at the Last Token With Information Transferred From Other Tokens
par: Mamidanna, Siddarth, et autres
Publié: (2025)

Robustness of Large Language Models to Perturbations in Text
par: Singh, Ayush, et autres
Publié: (2024)

Evaluating the Efficacy of Hybrid Deep Learning Models in Distinguishing AI-Generated Text
par: Oketunji, Abiodun Finbarrs
Publié: (2023)

Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy
par: He, Langzhou, et autres
Publié: (2026)

Eeyore: Realistic Depression Simulation via Supervised and Preference Optimization
par: Liu, Siyang, et autres
Publié: (2025)

Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective
par: Xu, Weijie, et autres
Publié: (2025)

Beyond Arabic: Software for Perso-Arabic Script Manipulation
par: Gutkin, Alexander, et autres
Publié: (2023)

Language Models are Crossword Solvers
par: Saha, Soumadeep, et autres
Publié: (2024)

Super Tiny Language Models
par: Hillier, Dylan, et autres
Publié: (2024)

Do Latent Tokens Think? A Causal and Adversarial Analysis of Chain-of-Continuous-Thought
par: Zhang, Yuyi, et autres
Publié: (2025)

Machine Translation Hallucination Detection for Low and High Resource Languages using Large Language Models
par: Benkirane, Kenza, et autres
Publié: (2024)

Textual Data Bias Detection and Mitigation -- An Extensible Pipeline with Experimental Evaluation
par: Görge, Rebekka, et autres
Publié: (2025)

Adaptive Focus Memory for Language Models
par: Cruz, Christopher
Publié: (2025)

Alignment Backfire: Language-Dependent Reversal of Safety Interventions Across 16 Languages in LLM Multi-Agent Systems
par: Fukui, Hiroki
Publié: (2026)