KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909101114523648 |
|---|---|
| author | Xu, Weijie Jiang, Xiaoyu Desai, Jay Han, Bin Yan, Fuqin Iannacci, Francis |
| author_facet | Xu, Weijie Jiang, Xiaoyu Desai, Jay Han, Bin Yan, Fuqin Iannacci, Francis |
| contents | In text classification tasks, fine tuning pretrained language models like BERT and GPT-3 yields competitive accuracy; however, both methods require pretraining on large text datasets. In contrast, general topic modeling methods possess the advantage of analyzing documents to extract meaningful patterns of words without the need of pretraining. To leverage topic modeling's unsupervised insights extraction on text classification tasks, we develop the Knowledge Distillation Semi-supervised Topic Modeling (KDSTM). KDSTM requires no pretrained embeddings, few labeled documents and is efficient to train, making it ideal under resource constrained settings. Across a variety of datasets, our method outperforms existing supervised topic modeling methods in classification accuracy, robustness and efficiency and achieves similar performance compare to state of the art weakly supervised text classification methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_01878 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation Xu, Weijie Jiang, Xiaoyu Desai, Jay Han, Bin Yan, Fuqin Iannacci, Francis Computation and Language Artificial Intelligence 68T50 I.2.6 In text classification tasks, fine tuning pretrained language models like BERT and GPT-3 yields competitive accuracy; however, both methods require pretraining on large text datasets. In contrast, general topic modeling methods possess the advantage of analyzing documents to extract meaningful patterns of words without the need of pretraining. To leverage topic modeling's unsupervised insights extraction on text classification tasks, we develop the Knowledge Distillation Semi-supervised Topic Modeling (KDSTM). KDSTM requires no pretrained embeddings, few labeled documents and is efficient to train, making it ideal under resource constrained settings. Across a variety of datasets, our method outperforms existing supervised topic modeling methods in classification accuracy, robustness and efficiency and achieves similar performance compare to state of the art weakly supervised text classification methods. |
| title | KDSTM: Neural Semi-supervised Topic Modeling with Knowledge Distillation |
| topic | Computation and Language Artificial Intelligence 68T50 I.2.6 |
| url | https://arxiv.org/abs/2307.01878 |