Improving Neural Topic Modeling with Semantically-Grounded Soft Label Distributions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912915145097216 |
|---|---|
| author | Li, Raymond Abaskohi, Amirhossein Li, Chuyuan Murray, Gabriel Carenini, Giuseppe |
| author_facet | Li, Raymond Abaskohi, Amirhossein Li, Chuyuan Murray, Gabriel Carenini, Giuseppe |
| contents | Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity. In this work, we propose a novel approach to construct semantically-grounded soft label targets using Language Models (LMs) by projecting the next token probabilities, conditioned on a specialized prompt, onto a pre-defined vocabulary to obtain contextually enriched supervision signals. By training the topic models to reconstruct the soft labels using the LM hidden states, our method produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus. Experiments on three datasets show that our method achieves substantial improvements in topic coherence, purity over existing baselines. Additionally, we also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents, highlighting its effectiveness for retrieval-oriented applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_17907 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Improving Neural Topic Modeling with Semantically-Grounded Soft Label Distributions Li, Raymond Abaskohi, Amirhossein Li, Chuyuan Murray, Gabriel Carenini, Giuseppe Computation and Language Artificial Intelligence Traditional neural topic models are typically optimized by reconstructing the document's Bag-of-Words (BoW) representations, overlooking contextual information and struggling with data sparsity. In this work, we propose a novel approach to construct semantically-grounded soft label targets using Language Models (LMs) by projecting the next token probabilities, conditioned on a specialized prompt, onto a pre-defined vocabulary to obtain contextually enriched supervision signals. By training the topic models to reconstruct the soft labels using the LM hidden states, our method produces higher-quality topics that are more closely aligned with the underlying thematic structure of the corpus. Experiments on three datasets show that our method achieves substantial improvements in topic coherence, purity over existing baselines. Additionally, we also introduce a retrieval-based metric, which shows that our approach significantly outperforms existing methods in identifying semantically similar documents, highlighting its effectiveness for retrieval-oriented applications. |
| title | Improving Neural Topic Modeling with Semantically-Grounded Soft Label Distributions |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2602.17907 |