Mask-guided BERT for Few Shot Text Classification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916482838953984 |
|---|---|
| author | Liao, Wenxiong Liu, Zhengliang Dai, Haixing Wu, Zihao Zhang, Yiyang Huang, Xiaoke Chen, Yuzhong Jiang, Xi Liu, Wei Zhu, Dajiang Liu, Tianming Li, Sheng Li, Xiang Cai, Hongmin |
| author_facet | Liao, Wenxiong Liu, Zhengliang Dai, Haixing Wu, Zihao Zhang, Yiyang Huang, Xiaoke Chen, Yuzhong Jiang, Xi Liu, Wei Zhu, Dajiang Liu, Tianming Li, Sheng Li, Xiang Cai, Hongmin |
| contents | Transformer-based language models have achieved significant success in various domains. However, the data-intensive nature of the transformer architecture requires much labeled data, which is challenging in low-resource scenarios (i.e., few-shot learning (FSL)). The main challenge of FSL is the difficulty of training robust models on small amounts of samples, which frequently leads to overfitting. Here we present Mask-BERT, a simple and modular framework to help BERT-based architectures tackle FSL. The proposed approach fundamentally differs from existing FSL strategies such as prompt tuning and meta-learning. The core idea is to selectively apply masks on text inputs and filter out irrelevant information, which guides the model to focus on discriminative tokens that influence prediction results. In addition, to make the text representations from different categories more separable and the text representations from the same category more compact, we introduce a contrastive learning loss function. Experimental results on public-domain benchmark datasets demonstrate the effectiveness of Mask-BERT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_10447 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Mask-guided BERT for Few Shot Text Classification Liao, Wenxiong Liu, Zhengliang Dai, Haixing Wu, Zihao Zhang, Yiyang Huang, Xiaoke Chen, Yuzhong Jiang, Xi Liu, Wei Zhu, Dajiang Liu, Tianming Li, Sheng Li, Xiang Cai, Hongmin Computation and Language Artificial Intelligence Transformer-based language models have achieved significant success in various domains. However, the data-intensive nature of the transformer architecture requires much labeled data, which is challenging in low-resource scenarios (i.e., few-shot learning (FSL)). The main challenge of FSL is the difficulty of training robust models on small amounts of samples, which frequently leads to overfitting. Here we present Mask-BERT, a simple and modular framework to help BERT-based architectures tackle FSL. The proposed approach fundamentally differs from existing FSL strategies such as prompt tuning and meta-learning. The core idea is to selectively apply masks on text inputs and filter out irrelevant information, which guides the model to focus on discriminative tokens that influence prediction results. In addition, to make the text representations from different categories more separable and the text representations from the same category more compact, we introduce a contrastive learning loss function. Experimental results on public-domain benchmark datasets demonstrate the effectiveness of Mask-BERT. |
| title | Mask-guided BERT for Few Shot Text Classification |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2302.10447 |