Cover Image

Optimal Embedding Learning Rate in LLMs: The Effect of Vocabulary Size

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hayou, Soufiane, Liu, Liyuan
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

A Proof of Learning Rate Transfer under $μ$P
by: Hayou, Soufiane
Published: (2025)

Learning Rate Scaling across LoRA Ranks and Transfer to Full Finetuning
by: Chen, Nan, et al.
Published: (2026)

The Impact of Initialization on LoRA Finetuning Dynamics
by: Hayou, Soufiane, et al.
Published: (2024)

LoRA+: Efficient Low Rank Adaptation of Large Models
by: Hayou, Soufiane, et al.
Published: (2024)

How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse
by: Seddik, Mohamed El Amine, et al.
Published: (2024)

PLoP: Precise LoRA Placement for Efficient Finetuning of Large Models
by: Hayou, Soufiane, et al.
Published: (2025)

Dynamic Vocabulary Pruning in Early-Exit LLMs
by: Vincenti, Jort, et al.
Published: (2024)

A Family of LLMs Liberated from Static Vocabularies
by: Alpha, Aleph, et al.
Published: (2026)

Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler
by: Shen, Yikang, et al.
Published: (2024)

Learning a Decision Tree Algorithm with Transformers
by: Zhuang, Yufan, et al.
Published: (2024)

Rule by Rule: Learning with Confidence through Vocabulary Expansion
by: Nössig, Albert, et al.
Published: (2024)

DOTS: Learning to Reason Dynamically in LLMs via Optimal Reasoning Trajectories Search
by: Yue, Murong, et al.
Published: (2024)

Dialogue Without Limits: Constant-Sized KV Caches for Extended Responses in LLMs
by: Ghadia, Ravi, et al.
Published: (2025)

Predictable Confabulations: Factual Recall by LLMs Scales with Model Size and Topic Frequency
by: Smith, Matthew L., et al.
Published: (2026)

Joint Embedding Learning of Educational Knowledge Graphs
by: Yao, Siyu, et al.
Published: (2019)

Text-ADBench: Text Anomaly Detection Benchmark based on LLMs Embedding
by: Xiao, Feng, et al.
Published: (2025)

Interpreting the Effects of Quantization on LLMs
by: Singh, Manpreet, et al.
Published: (2025)

Fast Vocabulary Transfer for Language Model Compression
by: Gee, Leonidas, et al.
Published: (2024)

$μ$pscaling small models: Principled warm starts and hyperparameter transfer
by: Ma, Yuxin, et al.
Published: (2026)

Provably Correct Automata Embeddings for Optimal Automata-Conditioned Reinforcement Learning
by: Yalcinkaya, Beyazit, et al.
Published: (2025)

LLMs as Zero-shot Graph Learners: Alignment of GNN Representations with LLM Token Embeddings
by: Wang, Duo, et al.
Published: (2024)

The Effect of Model Size on LLM Post-hoc Explainability via LIME
by: Heyen, Henning, et al.
Published: (2024)

CL4KGE: A Curriculum Learning Method for Knowledge Graph Embedding
by: Liu, Yang, et al.
Published: (2024)

Lossless Vocabulary Reduction for Auto-Regressive Language Models
by: Chijiwa, Daiki, et al.
Published: (2025)

Scaling Law with Learning Rate Annealing
by: Tissue, Howe, et al.
Published: (2024)

Integrating Time Series into LLMs via Multi-layer Steerable Embedding Fusion for Enhanced Forecasting
by: Chen, Zhuomin, et al.
Published: (2025)

T-FREE: Subword Tokenizer-Free Generative LLMs via Sparse Representations for Memory-Efficient Embeddings
by: Deiseroth, Björn, et al.
Published: (2024)

NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
by: Lee, Chankyu, et al.
Published: (2024)

Backward Lens: Projecting Language Model Gradients into the Vocabulary Space
by: Katz, Shahar, et al.
Published: (2024)

Grounded Token Initialization for New Vocabulary in LMs for Generative Recommendation
by: Chen, Daiwei, et al.
Published: (2026)

RobustSentEmbed: Robust Sentence Embeddings Using Adversarial Self-Supervised Contrastive Learning
by: Asl, Javad Rafiei, et al.
Published: (2024)

Factor Augmented Supervised Learning with Text Embeddings
by: Luo, Zhanye, et al.
Published: (2025)

LoRC: Low-Rank Compression for LLMs KV Cache with a Progressive Compression Strategy
by: Zhang, Rongzhi, et al.
Published: (2024)

Matryoshka Pilot: Learning to Drive Black-Box LLMs with LLMs
by: Li, Changhao, et al.
Published: (2024)

Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
by: Zheng, Chujie, et al.
Published: (2025)

Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies
by: Timor, Nadav, et al.
Published: (2025)

Small Vocabularies, Big Gains: Pretraining and Tokenization in Time Series Models
by: Roger, Alexis, et al.
Published: (2025)

Balancing Coverage and Draft Latency in Vocabulary Trimming for Faster Speculative Decoding
by: Shoham, Ofir Ben
Published: (2026)

Instruction Learning Paradigms: A Dual Perspective on White-box and Black-box LLMs
by: Ren, Yanwei, et al.
Published: (2025)

FR-Spec: Accelerating Large-Vocabulary Language Models via Frequency-Ranked Speculative Sampling
by: Zhao, Weilin, et al.
Published: (2025)