Cover Image

Tokens with Meaning: A Hybrid Tokenization Approach for Turkish

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Bayram, M. Ali, Fincan, Ali Arda, Gümüş, Ahmet Semih, Karakaş, Sercan, Diri, Banu, Yıldırım, Savaş, Çelik, Demircan
Format:	Preprint
Published:	2025
Subjects:	Computation and Language 68T50 I.2.7; I.2.6; H.3.1
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Tokenization Standards for Linguistic Integrity: Turkish as a Benchmark
by: Bayram, M. Ali, et al.
Published: (2025)

Doğal Dil İşlemede Tokenizasyon Standartları ve Ölçümü: Türkçe Üzerinden Büyük Dil Modellerinin Karşılaştırmalı Analizi
by: Bayram, M. Ali, et al.
Published: (2025)

Büyük Dil Modelleri için TR-MMLU Benchmarkı: Performans Değerlendirmesi, Zorluklar ve İyileştirme Fırsatları
by: Bayram, M. Ali, et al.
Published: (2025)

Setting Standards in Turkish NLP: TR-MMLU for Large Language Model Evaluation
by: Bayram, M. Ali, et al.
Published: (2024)

Low-Resource English-Tigrinya MT: Leveraging Multilingual Models, Custom Tokenizers, and Clean Evaluation Benchmarks
by: Teklehaymanot, Hailay Kidu, et al.
Published: (2025)

The Syntactic Acceptability Dataset (Preview): A Resource for Machine Learning and Linguistic Analysis of English
by: Juzek, Tom S
Published: (2025)

SampoNLP: A Self-Referential Toolkit for Morphological Analysis of Subword Tokenizers
by: Chelombitko, Iaroslav, et al.
Published: (2026)

IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following
by: Sun, Mingrui, et al.
Published: (2026)

Fine-tuning of Large Language Models for Constituency Parsing Using a Sequence to Sequence Approach
by: Delgado, Francisco Jose Cortes, et al.
Published: (2025)

Targeted Lexical Injection: Unlocking Latent Cross-Lingual Alignment in Lugha-Llama via Early-Layer LoRA Fine-Tuning
by: Ngugi, Stanley
Published: (2025)

How Human-Like Are Large Language Models? A Register-Aware Linguistic Evaluation Framework
by: Nieth, Björn, et al.
Published: (2026)

Survey Transfer Learning: Recycling Data with Silicon Responses
by: Amini, Ali
Published: (2025)

Future Token Prediction -- Causal Language Modelling with Per-Token Semantic State Vector for Multi-Token Prediction
by: Walker, Nicholas
Published: (2024)

IndiaFinBench: An Evaluation Benchmark for Large Language Model Performance on Indian Financial Regulatory Text
by: Pall, Rajveer Singh
Published: (2026)

When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models
by: Basu, Abhinaba
Published: (2026)

IntelliCode: A Multi-Agent LLM Tutoring System with Centralized Learner Modeling
by: David, Jones, et al.
Published: (2025)

LLMs as Architects and Critics for Multi-Source Opinion Summarization
by: Attri, Anuj, et al.
Published: (2025)

Why We Feel What We Feel: Joint Detection of Emotions and Their Opinion Triggers in E-commerce
by: Attri, Arnav, et al.
Published: (2025)

Word Overuse and Alignment in Large Language Models: The Influence of Learning from Human Feedback
by: Juzek, Tom S., et al.
Published: (2025)

Beyond Subtokens: A Rich Character Embedding for Low-resource and Morphologically Complex Languages
by: Schneider, Felix, et al.
Published: (2026)

From Prompting to Preference Optimization: A Comparative Study of LLM-based Automated Essay Scoring
by: Nguyen, Minh Hoang, et al.
Published: (2026)

Before the Last Token: Diagnosing Final-Token Safety Probe Failures
by: Doda, Shravan
Published: (2026)

Constitution or Collapse? Exploring Constitutional AI with Llama 3-8B
by: Zhang, Xue
Published: (2025)

Learning What Matters: Probabilistic Task Selection via Mutual Information for Model Finetuning
by: Chanda, Prateek, et al.
Published: (2025)

XAutoLM: Efficient Fine-Tuning of Language Models via Meta-Learning and AutoML
by: Estevanell-Valladares, Ernesto L., et al.
Published: (2025)

How much do LLMs learn from negative examples?
by: Hamdan, Shadi, et al.
Published: (2025)

Communicative Agents for Slideshow Storytelling Video Generation based on LLMs
by: Fan, Jingxing, et al.
Published: (2025)

Improving Commonsense Bias Classification by Mitigating the Influence of Demographic Terms
by: Lee, JinKyu, et al.
Published: (2024)

Causally Grounded Mechanistic Interpretability for LLMs with Faithful Natural-Language Explanations
by: Mahale, Ajay Pravin
Published: (2026)

From Noise to Diversity: Random Embedding Injection in LLM Reasoning
by: Kim, Heejun, et al.
Published: (2026)

A Confidence-Diversity Framework for Calibrating AI Judgement in Accessible Qualitative Coding Tasks
by: Zhao, Zhilong, et al.
Published: (2025)

NOTAI.AI: Explainable Detection of Machine-Generated Text via Curvature and Feature Attribution
by: Breneur, Oleksandr Marchenko, et al.
Published: (2026)

Semantic Convergence: Investigating Shared Representations Across Scaled LLMs
by: Son, Daniel, et al.
Published: (2025)

Rethinking the Multilingual Reasoning Gap with Layer Swap
by: Lasbordes, Maxence, et al.
Published: (2026)

Towards Ontology-Enhanced Representation Learning for Large Language Models
by: Ronzano, Francesco, et al.
Published: (2024)

Improving Large-Scale k-Nearest Neighbor Text Categorization with Label Autoencoders
by: Ribadas-Pena, Francisco J., et al.
Published: (2024)

LLM-supported document separation for printed reviews from zbMATH Open
by: Pluzhnikov, Ivan, et al.
Published: (2026)

Pre-trained Models Perform the Best When Token Distributions Follow Zipf's Law
by: He, Yanjin, et al.
Published: (2025)

Kronecker Embeddings: Byte-Level Structured Token Representations for Parameter-Efficient Language Models
by: Shravan, Rohan
Published: (2026)

ProofBridge: Auto-Formalization of Natural Language Proofs in Lean via Joint Embeddings
by: Jana, Prithwish, et al.
Published: (2025)