A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917827957489664 |
|---|---|
| author | Ackerman, Samuel Rabinovich, Ella Farchi, Eitan Anaby-Tavor, Ateret |
| author_facet | Ackerman, Samuel Rabinovich, Ella Farchi, Eitan Anaby-Tavor, Ateret |
| contents | We evaluate the robustness of several large language models on multiple datasets. Robustness here refers to the relative insensitivity of the model's answers to meaning-preserving variants of their input. Benchmark datasets are constructed by introducing naturally-occurring, non-malicious perturbations, or by generating semantically equivalent paraphrases of input questions or statements. We further propose a novel metric for assessing a model robustness, and demonstrate its benefits in the non-adversarial scenario by empirical evaluation of several models on the created datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01963 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios Ackerman, Samuel Rabinovich, Ella Farchi, Eitan Anaby-Tavor, Ateret Computation and Language Applications We evaluate the robustness of several large language models on multiple datasets. Robustness here refers to the relative insensitivity of the model's answers to meaning-preserving variants of their input. Benchmark datasets are constructed by introducing naturally-occurring, non-malicious perturbations, or by generating semantically equivalent paraphrases of input questions or statements. We further propose a novel metric for assessing a model robustness, and demonstrate its benefits in the non-adversarial scenario by empirical evaluation of several models on the created datasets. |
| title | A Novel Metric for Measuring the Robustness of Large Language Models in Non-adversarial Scenarios |
| topic | Computation and Language Applications |
| url | https://arxiv.org/abs/2408.01963 |