Cover Image

Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Tao, Chaofan, Liu, Qian, Dou, Longxu, Muennighoff, Niklas, Wan, Zhongwei, Luo, Ping, Lin, Min, Wong, Ngai
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

RegMix: Data Mixture as Regression for Language Model Pre-training
by: Liu, Qian, et al.
Published: (2024)

Seamless Deception: Larger Language Models Are Better Knowledge Concealers
by: Ashok, Dhananjay, et al.
Published: (2026)

Sailor: Open Language Models for South-East Asia
by: Dou, Longxu, et al.
Published: (2024)

Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling
by: Shin, Haebin, et al.
Published: (2025)

Why Larger Language Models Do In-context Learning Differently?
by: Shi, Zhenmei, et al.
Published: (2024)

Distill Not Only Data but Also Rewards: Can Smaller Language Models Surpass Larger Ones?
by: Zhang, Yudi, et al.
Published: (2025)

Larger or Smaller Reward Margins to Select Preferences for Alignment?
by: Huang, Kexin, et al.
Published: (2025)

Surprising Efficacy of Fine-Tuned Transformers for Fact-Checking over Larger Language Models
by: Setty, Vinay
Published: (2024)

Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning
by: Juneja, Gurusha, et al.
Published: (2023)

Rethinking Kullback-Leibler Divergence in Knowledge Distillation for Large Language Models
by: Wu, Taiqiang, et al.
Published: (2024)

DVAGen: Dynamic Vocabulary Augmented Generation
by: Du, Wei, et al.
Published: (2025)

Scaling LLM Pre-training with Vocabulary Curriculum
by: Yu, Fangyuan
Published: (2025)

Reasoning Does Not Necessarily Improve Role-Playing Ability
by: Feng, Xiachong, et al.
Published: (2025)

Exploring the Knowledge Mismatch Hypothesis: Hallucination Propensity in Small Models Fine-tuned on Data from Larger Models
by: Wee, Phil, et al.
Published: (2024)

Addressing the Ecological Fallacy in Larger LMs with Human Context
by: Soni, Nikita, et al.
Published: (2026)

Larger Language Models Don't Care How You Think: Why Chain-of-Thought Prompting Fails in Subjective Tasks
by: Chochlakis, Georgios, et al.
Published: (2024)

The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
by: Hassid, Michael, et al.
Published: (2024)

Revisiting Model Interpolation for Efficient Reasoning
by: Wu, Taiqiang, et al.
Published: (2025)

Larger models yield better results? Streamlined severity classification of ADHD-related concerns using BERT-based knowledge distillation
by: Karim, Ahmed Akib Jawad, et al.
Published: (2024)

Optimal Turkish Subword Strategies at Scale: Systematic Evaluation of Data, Vocabulary, Morphology Interplay
by: Altinok, Duygu
Published: (2026)

Fast Vocabulary Transfer for Language Model Compression
by: Gee, Leonidas, et al.
Published: (2024)

A Multitask Training Approach to Enhance Whisper with Contextual Biasing and Open-Vocabulary Keyword Spotting
by: Li, Yuang, et al.
Published: (2023)

CSV-Decode: Certifiable Sub-Vocabulary Decoding for Efficient Large Language Model Inference
by: Liu, Dong, et al.
Published: (2025)

Big Tech-Funded AI Papers Have Higher Citation Impact, Greater Insularity, and Larger Recency Bias
by: Gnewuch, Max Martin, et al.
Published: (2025)

Efficient and Effective Vocabulary Expansion Towards Multilingual Large Language Models
by: Kim, Seungduk, et al.
Published: (2024)

The Scandinavian Embedding Benchmarks: Comprehensive Assessment of Multilingual and Monolingual Text Embedding
by: Enevoldsen, Kenneth, et al.
Published: (2024)

Lossless Vocabulary Reduction for Auto-Regressive Language Models
by: Chijiwa, Daiki, et al.
Published: (2025)

How Vocabulary Sharing Facilitates Multilingualism in LLaMA?
by: Yuan, Fei, et al.
Published: (2023)

Vocabulary Dropout for Curriculum Diversity in LLM Co-Evolution
by: Dineen, Jacob, et al.
Published: (2026)

Using Perspectival Words Is Harder Than Vocabulary Words for Humans and Even More So for Multimodal Language Models
by: Dong, Dota Tianai, et al.
Published: (2025)

An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference
by: Yamaguchi, Atsuki, et al.
Published: (2024)

Uniform Memory Retrieval with Larger Capacity for Modern Hopfield Models
by: Wu, Dennis, et al.
Published: (2024)

Timber: Training-free Instruct Model Refining with Base via Effective Rank
by: Wu, Taiqiang, et al.
Published: (2025)

Training Optimal Large Diffusion Language Models
by: Ni, Jinjie, et al.
Published: (2025)

A Zero-Shot Open-Vocabulary Pipeline for Dialogue Understanding
by: Safa, Abdulfattah, et al.
Published: (2024)

Column Vocabulary Association (CVA): semantic interpretation of dataless tables
by: Martorana, Margherita, et al.
Published: (2024)

We Can't Understand AI Using our Existing Vocabulary
by: Hewitt, John, et al.
Published: (2025)

Happiness is Sharing a Vocabulary: A Study of Transliteration Methods
by: Jung, Haeji, et al.
Published: (2025)

Now It Sounds Like You: Learning Personalized Vocabulary On Device
by: Wang, Sid, et al.
Published: (2023)

Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size
by: Hayou, Soufiane, et al.
Published: (2025)