Imagen de Portada

How does a Language-Specific Tokenizer affect LLMs?

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Seo, Jean, Kim, Jaeyoon, Byun, SungJoo, Shin, Hyopil
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

MoFE: Mixture of Frozen Experts Architecture
por: Seo, Jean, et al.
Publicado: (2025)

A Study on How Attention Scores in the BERT Model are Aware of Lexical Categories in Syntactic and Semantic Tasks on the GLUE Benchmark
por: Jang, Dongjun, et al.
Publicado: (2024)

CARBD-Ko: A Contextually Annotated Review Benchmark Dataset for Aspect-Level Sentiment Classification in Korean
por: Jang, Dongjun, et al.
Publicado: (2024)

KIT-19: A Comprehensive Korean Instruction Toolkit on 19 Tasks for Fine-Tuning Korean Large Language Models
por: Jang, Dongjun, et al.
Publicado: (2024)

DAHL: Domain-specific Automated Hallucination Evaluation of Long-Form Text through a Benchmark Dataset in Biomedicine
por: Seo, Jean, et al.
Publicado: (2024)

Korean Bio-Medical Corpus (KBMC) for Medical Named Entity Recognition
por: Byun, Sungjoo, et al.
Publicado: (2024)

P-CoT: A Pedagogically-motivated Participatory Chain-of-Thought Prompting for Phonological Reasoning in LLMs
por: Jang, Dongjun, et al.
Publicado: (2025)

RCScore: Quantifying Response Consistency in Large Language Models
por: Jang, Dongjun, et al.
Publicado: (2025)

How Language Directions Align with Token Geometry in Multilingual LLMs
por: Kim, JaeSeong, et al.
Publicado: (2025)

KOFFVQA: An Objectively Evaluated Free-form VQA Benchmark for Large Vision-Language Models in the Korean Language
por: Kim, Yoonshik, et al.
Publicado: (2025)

KoBALT: Korean Benchmark For Advanced Linguistic Tasks
por: Shin, Hyopil, et al.
Publicado: (2025)

LLaVA-Docent: Instruction Tuning with Multimodal Large Language Model to Support Art Appreciation Education
por: Lee, Unggi, et al.
Publicado: (2024)

How Well Do Large Language Models Truly Ground?
por: Lee, Hyunji, et al.
Publicado: (2023)

Beyond Early-Token Bias: Model-Specific and Language-Specific Position Effects in Multilingual LLMs
por: Menschikov, Mikhail, et al.
Publicado: (2025)

Mergen: The First Manchu-Korean Machine Translation Model Trained on Augmented Data
por: Seo, Jean, et al.
Publicado: (2023)

Semiparametric Token-Sequence Co-Supervision
por: Lee, Hyunji, et al.
Publicado: (2024)

Measuring Inclusion in Interaction: Inclusion Analytics for Human-AI Collaborative Learning
por: Choi, Jaeyoon, et al.
Publicado: (2026)

How Chinese are Chinese Language Models? The Puzzling Lack of Language Policy in China's LLMs
por: Wen-Yi, Andrea W, et al.
Publicado: (2024)

Optimizing Korean-Centric LLMs via Token Pruning
por: Kim, Hoyeol, et al.
Publicado: (2026)

DNACHUNKER: Learnable Tokenization for DNA Language Models
por: Kim, Taewon, et al.
Publicado: (2026)

ManWav: The First Manchu ASR Model
por: Seo, Jean, et al.
Publicado: (2024)

HerO at AVeriTeC: The Herd of Open Large Language Models for Verifying Real-World Claims
por: Yoon, Yejun, et al.
Publicado: (2024)

How does a Multilingual LM Handle Multiple Languages?
por: Kakarla, Santhosh, et al.
Publicado: (2025)

Exploring Fine-Tuning of Large Audio Language Models for Spoken Language Understanding under Limited Speech Data
por: Choi, Youngwon, et al.
Publicado: (2025)

Evaluating the Pre-Consultation Ability of LLMs using Diagnostic Guidelines
por: Seo, Jean, et al.
Publicado: (2026)

SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs
por: Aditi, et al.
Publicado: (2025)

LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs
por: An, Sumin, et al.
Publicado: (2025)

How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?
por: Lee, Seongyun, et al.
Publicado: (2024)

How Tokenization Limits Phonological Knowledge Representation in Language Models and How to Improve Them
por: Liao, Disen, et al.
Publicado: (2026)

How does Misinformation Affect Large Language Model Behaviors and Preferences?
por: Peng, Miao, et al.
Publicado: (2025)

Fine-Grained and Thematic Evaluation of LLMs in Social Deduction Game
por: Kim, Byungjun, et al.
Publicado: (2024)

Incorporating Domain Knowledge into Materials Tokenization
por: Oh, Yerim, et al.
Publicado: (2025)

An Implementation of Werewolf Agent That does not Truly Trust LLMs
por: Sato, Takehiro, et al.
Publicado: (2024)

Is a Picture Worth a Thousand Words? Adaptive Multimodal Fact-Checking with Visual Evidence Necessity
por: Jung, Jaeyoon, et al.
Publicado: (2026)

VILLAIN at AVerImaTeC: Verifying Image-Text Claims via Multi-Agent Collaboration
por: Jung, Jaeyoon, et al.
Publicado: (2026)

On Efficient Language and Vision Assistants for Visually-Situated Natural Language Understanding: What Matters in Reading and Reasoning
por: Kim, Geewook, et al.
Publicado: (2024)

Training Text-to-Molecule Models with Context-Aware Tokenization
por: Kim, Seojin, et al.
Publicado: (2025)

Measuring Sycophancy of Language Models in Multi-turn Dialogues
por: Hong, Jiseung, et al.
Publicado: (2025)

Team HUMANE at AVeriTeC 2025: HerO 2 for Efficient Fact Verification
por: Yoon, Yejun, et al.
Publicado: (2025)

RSQ: Learning from Important Tokens Leads to Better Quantized LLMs
por: Sung, Yi-Lin, et al.
Publicado: (2025)