Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.13434 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915292823683072 |
|---|---|
| author | Bystroński, Mateusz Hołysz, Mikołaj Piotrowski, Grzegorz Chawla, Nitesh V. Kajdanowicz, Tomasz |
| author_facet | Bystroński, Mateusz Hołysz, Mikołaj Piotrowski, Grzegorz Chawla, Nitesh V. Kajdanowicz, Tomasz |
| contents | Data scarcity and class imbalance are persistent challenges in training robust NLP models, especially in specialized domains or low-resource settings. We propose a novel technique, SMOTExT, that adapts the idea of Synthetic Minority Over-sampling (SMOTE) to textual data. Our method generates new synthetic examples by interpolating between BERT-based embeddings of two existing examples and then decoding the resulting latent point into text with xRAG architecture. By leveraging xRAG's cross-modal retrieval-generation framework, we can effectively turn interpolated vectors into coherent text. While this is preliminary work supported by qualitative outputs only, the method shows strong potential for knowledge distillation and data augmentation in few-shot settings. Notably, our approach also shows promise for privacy-preserving machine learning: in early experiments, training models solely on generated data achieved comparable performance to models trained on the original dataset. This suggests a viable path toward safe and effective learning under data protection constraints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_13434 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SMOTExT: SMOTE meets Large Language Models Bystroński, Mateusz Hołysz, Mikołaj Piotrowski, Grzegorz Chawla, Nitesh V. Kajdanowicz, Tomasz Computation and Language Data scarcity and class imbalance are persistent challenges in training robust NLP models, especially in specialized domains or low-resource settings. We propose a novel technique, SMOTExT, that adapts the idea of Synthetic Minority Over-sampling (SMOTE) to textual data. Our method generates new synthetic examples by interpolating between BERT-based embeddings of two existing examples and then decoding the resulting latent point into text with xRAG architecture. By leveraging xRAG's cross-modal retrieval-generation framework, we can effectively turn interpolated vectors into coherent text. While this is preliminary work supported by qualitative outputs only, the method shows strong potential for knowledge distillation and data augmentation in few-shot settings. Notably, our approach also shows promise for privacy-preserving machine learning: in early experiments, training models solely on generated data achieved comparable performance to models trained on the original dataset. This suggests a viable path toward safe and effective learning under data protection constraints. |
| title | SMOTExT: SMOTE meets Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.13434 |