Lightweight Contenders: Navigating Semi-Supervised Text Mining through Peer Collaboration and Self Transcendence
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915042540126208 |
|---|---|
| author | Mao, Qianren Jiang, Weifeng Liu, Junnan Lin, Chenghua Li, Qian Wen, Xianqing Li, Jianxin Lu, Jinhu |
| author_facet | Mao, Qianren Jiang, Weifeng Liu, Junnan Lin, Chenghua Li, Qian Wen, Xianqing Li, Jianxin Lu, Jinhu |
| contents | The semi-supervised learning (SSL) strategy in lightweight models requires reducing annotated samples and facilitating cost-effective inference. However, the constraint on model parameters, imposed by the scarcity of training labels, limits the SSL performance. In this paper, we introduce PS-NET, a novel framework tailored for semi-supervised text mining with lightweight models. PS-NET incorporates online distillation to train lightweight student models by imitating the Teacher model. It also integrates an ensemble of student peers that collaboratively instruct each other. Additionally, PS-NET implements a constant adversarial perturbation schema to further self-augmentation by progressive generalizing. Our PS-NET, equipped with a 2-layer distilled BERT, exhibits notable performance enhancements over SOTA lightweight SSL frameworks of FLiText and DisCo in SSL text classification with extremely rare labelled data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_00883 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Lightweight Contenders: Navigating Semi-Supervised Text Mining through Peer Collaboration and Self Transcendence Mao, Qianren Jiang, Weifeng Liu, Junnan Lin, Chenghua Li, Qian Wen, Xianqing Li, Jianxin Lu, Jinhu Computation and Language Machine Learning The semi-supervised learning (SSL) strategy in lightweight models requires reducing annotated samples and facilitating cost-effective inference. However, the constraint on model parameters, imposed by the scarcity of training labels, limits the SSL performance. In this paper, we introduce PS-NET, a novel framework tailored for semi-supervised text mining with lightweight models. PS-NET incorporates online distillation to train lightweight student models by imitating the Teacher model. It also integrates an ensemble of student peers that collaboratively instruct each other. Additionally, PS-NET implements a constant adversarial perturbation schema to further self-augmentation by progressive generalizing. Our PS-NET, equipped with a 2-layer distilled BERT, exhibits notable performance enhancements over SOTA lightweight SSL frameworks of FLiText and DisCo in SSL text classification with extremely rare labelled data. |
| title | Lightweight Contenders: Navigating Semi-Supervised Text Mining through Peer Collaboration and Self Transcendence |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2412.00883 |