Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.14060 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912906191306752 |
|---|---|
| author | Liu, Yang Yang, Jiaye Li, Weikang Liang, Jiahui Li, Yang Yan, Lingyong |
| author_facet | Liu, Yang Yang, Jiaye Li, Weikang Liang, Jiahui Li, Yang Yan, Lingyong |
| contents | We introduce LM-Lexicon, an innovative definition modeling approach that incorporates data clustering, semantic expert learning, and model merging using a sparse mixture-of-experts architecture. By decomposing the definition modeling task into specialized semantic domains, where small language models are trained as domain experts, LM-Lexicon achieves substantial improvements (+7% BLEU score compared with the prior state-of-the-art model) over existing methods on five widely used benchmarks. Empirically, we demonstrate that 1) the clustering strategy enables fine-grained expert specialization with nearly 10% improvement in definition quality; 2) the semantic-aware domain-level routing mechanism achieves higher expert efficacy (+1%) than conventional token-level routing; and 3) further performance gains can be obtained through test-time compute and semantic expert scaling. Our work advances definition modeling while providing insights into the development of efficient language models for semantic-intensive applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_14060 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts Liu, Yang Yang, Jiaye Li, Weikang Liang, Jiahui Li, Yang Yan, Lingyong Computation and Language We introduce LM-Lexicon, an innovative definition modeling approach that incorporates data clustering, semantic expert learning, and model merging using a sparse mixture-of-experts architecture. By decomposing the definition modeling task into specialized semantic domains, where small language models are trained as domain experts, LM-Lexicon achieves substantial improvements (+7% BLEU score compared with the prior state-of-the-art model) over existing methods on five widely used benchmarks. Empirically, we demonstrate that 1) the clustering strategy enables fine-grained expert specialization with nearly 10% improvement in definition quality; 2) the semantic-aware domain-level routing mechanism achieves higher expert efficacy (+1%) than conventional token-level routing; and 3) further performance gains can be obtained through test-time compute and semantic expert scaling. Our work advances definition modeling while providing insights into the development of efficient language models for semantic-intensive applications. |
| title | LM-Lexicon: Improving Definition Modeling via Harmonizing Semantic Experts |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.14060 |