Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910966512353280 |
|---|---|
| author | Zhang, Junyan Huang, Yiming Liu, Shuliang Gao, Yubo Hu, Xuming |
| author_facet | Zhang, Junyan Huang, Yiming Liu, Shuliang Gao, Yubo Hu, Xuming |
| contents | The rapid adoption of LLMs has overshadowed the potential advantages of traditional BERT-like models in text classification. This study challenges the prevailing "LLM-centric" trend by systematically comparing three category methods, i.e., BERT-like models fine-tuning, LLM internal state utilization, and zero-shot inference across six high-difficulty datasets. Our findings reveal that BERT-like models often outperform LLMs. We further categorize datasets into three types, perform PCA and probing experiments, and identify task-specific model strengths: BERT-like models excel in pattern-driven tasks, while LLMs dominate those requiring deep semantics or world knowledge. Based on this, we propose TaMAS, a fine-grained task selection strategy, advocating for a nuanced, task-driven approach over a one-size-fits-all reliance on LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18215 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs? Zhang, Junyan Huang, Yiming Liu, Shuliang Gao, Yubo Hu, Xuming Computation and Language Artificial Intelligence The rapid adoption of LLMs has overshadowed the potential advantages of traditional BERT-like models in text classification. This study challenges the prevailing "LLM-centric" trend by systematically comparing three category methods, i.e., BERT-like models fine-tuning, LLM internal state utilization, and zero-shot inference across six high-difficulty datasets. Our findings reveal that BERT-like models often outperform LLMs. We further categorize datasets into three types, perform PCA and probing experiments, and identify task-specific model strengths: BERT-like models excel in pattern-driven tasks, while LLMs dominate those requiring deep semantics or world knowledge. Based on this, we propose TaMAS, a fine-grained task selection strategy, advocating for a nuanced, task-driven approach over a one-size-fits-all reliance on LLMs. |
| title | Do BERT-Like Bidirectional Models Still Perform Better on Text Classification in the Era of LLMs? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2505.18215 |