Copertina

Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Cho, Gyeongje, So, Yeonkyoun, Park, Chanwoo, Lee, Sangmin, Jung, Sungmok, Lee, Jaejin
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Computation and Language Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Thunder-LLM: Efficiently Adapting LLMs to Korean with Minimal Resources
di: Kim, Jinpyo, et al.
Pubblicazione: (2025)

Choices Speak Louder than Questions
di: Cho, Gyeongje, et al.
Pubblicazione: (2025)

Thunder-KoNUBench: A Corpus-Aligned Benchmark for Korean Negation Understanding
di: Jung, Sungmok, et al.
Pubblicazione: (2026)

Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding
di: So, Yeonkyoung, et al.
Pubblicazione: (2025)

Models Know Models Best: Evaluation via Model-Preferred Formats
di: Lee, Joonhak, et al.
Pubblicazione: (2026)

Accelerating Multilingual Language Model for Excessively Tokenized Languages
di: Hong, Jimin, et al.
Pubblicazione: (2024)

Ko-MuSR: A Multistep Soft Reasoning Benchmark for LLMs Capable of Understanding Korean
di: Park, Chanwoo, et al.
Pubblicazione: (2025)

Beyond Line-Level Filtering for the Pretraining Corpora of LLMs
di: Park, Chanwoo, et al.
Pubblicazione: (2025)

Thunder-DeID: Accurate and Efficient De-identification Framework for Korean Court Judgments
di: Hahm, Sungeun, et al.
Pubblicazione: (2025)

TiTok: Transfer Token-level Knowledge via Contrastive Excess to Transplant LoRA
di: Jung, Chanjoo, et al.
Pubblicazione: (2025)

SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance
di: Tănase, Andrei-Valentin, et al.
Pubblicazione: (2025)

TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoning
di: Zhang, Tunyu, et al.
Pubblicazione: (2025)

FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs
di: Lee, Shinbok, et al.
Pubblicazione: (2024)

Speed and Conversational Large Language Models: Not All Is About Tokens per Second
di: Conde, Javier, et al.
Pubblicazione: (2025)

Open Ko-LLM Leaderboard: Evaluating Large Language Models in Korean with Ko-H5 Benchmark
di: Park, Chanjun, et al.
Pubblicazione: (2024)

DyLLM: Efficient Diffusion LLM Inference via Saliency-based Token Selection and Partial Attention
di: Lee, Younjoo, et al.
Pubblicazione: (2026)

Token-Supervised Value Models for Enhancing Mathematical Problem-Solving Capabilities of Large Language Models
di: Lee, Jung Hyun, et al.
Pubblicazione: (2024)

From Tokens to Words: On the Inner Lexicon of LLMs
di: Kaplan, Guy, et al.
Pubblicazione: (2024)

Text Generation Beyond Discrete Token Sampling
di: Zhuang, Yufan, et al.
Pubblicazione: (2025)

TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation
di: Lin, Haokun, et al.
Pubblicazione: (2025)

Incorporating Domain Knowledge into Materials Tokenization
di: Oh, Yerim, et al.
Pubblicazione: (2025)

Theme-Explanation Structure for Table Summarization using Large Language Models: A Case Study on Korean Tabular Data
di: Kwack, TaeYoon, et al.
Pubblicazione: (2025)

Control Token with Dense Passage Retrieval
di: Lee, Juhwan, et al.
Pubblicazione: (2024)

TimeTok: Granularity-Controllable Time-Series Generation via Hierarchical Tokenization
di: Lee, Seokhyun, et al.
Pubblicazione: (2026)

Unsupervised Extractive Dialogue Summarization in Hyperdimensional Space
di: Park, Seongmin, et al.
Pubblicazione: (2024)

The Art of Breaking Words: Rethinking Multilingual Tokenizer Design
di: Thakur, Aamod, et al.
Pubblicazione: (2025)

UKTA: Unified Korean Text Analyzer
di: Ahn, Seokho, et al.
Pubblicazione: (2025)

Thinking Tokens for Language Modeling
di: Herel, David, et al.
Pubblicazione: (2024)

Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization
di: Wang, Dixuan, et al.
Pubblicazione: (2024)

Explainability-Based Token Replacement on LLM-Generated Text
di: Mohammadi, Hadi, et al.
Pubblicazione: (2025)

On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models
di: Seo, Hoigi, et al.
Pubblicazione: (2025)

Building Resource-Constrained Language Agents: A Korean Case Study on Chemical Toxicity Information
di: Cho, Hojun, et al.
Pubblicazione: (2025)

On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation
di: Hsu, Chan-Jan, et al.
Pubblicazione: (2026)

SemToken: Semantic-Aware Tokenization for Efficient Long-Context Language Modeling
di: Liu, Dong, et al.
Pubblicazione: (2025)

One Token Is Enough: Improving Diffusion Language Models with a Sink Token
di: Zhang, Zihou, et al.
Pubblicazione: (2026)

Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning
di: Su, DiJia, et al.
Pubblicazione: (2025)

Training Text-to-Molecule Models with Context-Aware Tokenization
di: Kim, Seojin, et al.
Pubblicazione: (2025)

SCRIPT: A Subcharacter Compositional Representation Injection Module for Korean Pre-Trained Language Models
di: Kim, SungHo, et al.
Pubblicazione: (2026)

Alternatives To Next Token Prediction In Text Generation -- A Survey
di: Wyatt, Charlie, et al.
Pubblicazione: (2025)

Reinforcement Learning with Token-level Feedback for Controllable Text Generation
di: Li, Wendi, et al.
Pubblicazione: (2024)