Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2602.00217 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911645485236224 |
|---|---|
| author | Liu, Chen Sun, Xingzhi Xiao, Xi Van Tassel, Alexandre Xu, Ke Reimann, Kristof Liao, Danqi Gerstein, Mark Wang, Tianyang Wang, Xiao Krishnaswamy, Smita |
| author_facet | Liu, Chen Sun, Xingzhi Xiao, Xi Van Tassel, Alexandre Xu, Ke Reimann, Kristof Liao, Danqi Gerstein, Mark Wang, Tianyang Wang, Xiao Krishnaswamy, Smita |
| contents | Large language models (LLMs) achieve remarkable performance through ever-increasing parameter counts, but scaling incurs steep computational costs. To better understand LLM scaling, we study representational differences between LLMs and their smaller counterparts, with the goal of replicating the representational qualities of larger models in smaller models. We observe a geometric phenomenon which we term $\textbf{embedding condensation}$, where token embeddings collapse into a narrow cone-like subspace in some language models. Through systematic analyses across multiple Transformer families, we show that small models such as $\texttt{GPT2}$ and $\texttt{Qwen3-0.6B}$ exhibit severe condensation, whereas larger models such as $\texttt{GPT2-xl}$ and $\texttt{Qwen3-32B}$ are more resistant to this phenomenon. Additional observations show that embedding condensation is not reliably mitigated by knowledge distillation from larger models. To fight against it, we formulate a dispersion loss that explicitly encourages embedding dispersion during training. Experiments demonstrate that it mitigates condensation, recovers dispersion patterns seen in larger models, and yields performance gains across 10 benchmarks. We believe this work offers a principled path toward improving smaller Transformers without additional parameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_00217 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models Liu, Chen Sun, Xingzhi Xiao, Xi Van Tassel, Alexandre Xu, Ke Reimann, Kristof Liao, Danqi Gerstein, Mark Wang, Tianyang Wang, Xiao Krishnaswamy, Smita Machine Learning Large language models (LLMs) achieve remarkable performance through ever-increasing parameter counts, but scaling incurs steep computational costs. To better understand LLM scaling, we study representational differences between LLMs and their smaller counterparts, with the goal of replicating the representational qualities of larger models in smaller models. We observe a geometric phenomenon which we term $\textbf{embedding condensation}$, where token embeddings collapse into a narrow cone-like subspace in some language models. Through systematic analyses across multiple Transformer families, we show that small models such as $\texttt{GPT2}$ and $\texttt{Qwen3-0.6B}$ exhibit severe condensation, whereas larger models such as $\texttt{GPT2-xl}$ and $\texttt{Qwen3-32B}$ are more resistant to this phenomenon. Additional observations show that embedding condensation is not reliably mitigated by knowledge distillation from larger models. To fight against it, we formulate a dispersion loss that explicitly encourages embedding dispersion during training. Experiments demonstrate that it mitigates condensation, recovers dispersion patterns seen in larger models, and yields performance gains across 10 benchmarks. We believe this work offers a principled path toward improving smaller Transformers without additional parameters. |
| title | Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.00217 |