Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913444081434624 |
|---|---|
| author | Stacey, Joe Rei, Marek |
| author_facet | Stacey, Joe Rei, Marek |
| contents | Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some of the performance benefits. While this method can improve results on in-distribution examples, it does not necessarily generalise to out-of-distribution (OOD) settings. We investigate two complementary methods for improving the robustness of the resulting student models on OOD domains. The first approach augments the distillation with generated unlabelled examples that match the target distribution. The second method upsamples data points among the training set that are similar to the target distribution. When applied on the task of natural language inference (NLI), our experiments on MNLI show that distillation with these modifications outperforms previous robustness solutions. We also find that these methods improve performance on OOD domains even beyond the target domain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_13067 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation Stacey, Joe Rei, Marek Computation and Language Machine Learning I.2.7 Knowledge distillation optimises a smaller student model to behave similarly to a larger teacher model, retaining some of the performance benefits. While this method can improve results on in-distribution examples, it does not necessarily generalise to out-of-distribution (OOD) settings. We investigate two complementary methods for improving the robustness of the resulting student models on OOD domains. The first approach augments the distillation with generated unlabelled examples that match the target distribution. The second method upsamples data points among the training set that are similar to the target distribution. When applied on the task of natural language inference (NLI), our experiments on MNLI show that distillation with these modifications outperforms previous robustness solutions. We also find that these methods improve performance on OOD domains even beyond the target domain. |
| title | Distilling Robustness into Natural Language Inference Models with Domain-Targeted Augmentation |
| topic | Computation and Language Machine Learning I.2.7 |
| url | https://arxiv.org/abs/2305.13067 |