FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917283426729984 |
|---|---|
| author | Otmakhova, Yulia Truong, Hung Thinh Mahendra, Rahmad Zhai, Zenan Zhu, Rongxin Beck, Daniel Lau, Jey Han |
| author_facet | Otmakhova, Yulia Truong, Hung Thinh Mahendra, Rahmad Zhai, Zenan Zhu, Rongxin Beck, Daniel Lau, Jey Han |
| contents | We present FLUKE (Framework for LingUistically-driven and tasK-agnostic robustness Evaluation), a framework for assessing model robustness through systematic minimal variations of test data. FLUKE introduces controlled variations across linguistic levels -- from orthography to dialect and style -- and leverages large language models (LLMs) with human validation to generate modifications. We demonstrate FLUKE's utility by evaluating both fine-tuned models and LLMs across six diverse NLP tasks (four classification and two generation tasks), and reveal that (1) the impact of linguistic variations is highly task-dependent, with some tests being critical for certain tasks but irrelevant for others; (2) LLMs still exhibit significant brittleness to certain linguistic variations, with reasoning LLMs surprisingly showing less robustness on some tasks compared to base models, and scaling improving robustness only for surface-level modifications; (3) models are overall more brittle to natural, fluent modifications such as syntax or style changes (and especially to negation), compared to corruption-style tests such as letter flipping; (4) the ability of a model to use a linguistic feature in generation does not correlate to its robustness to this feature on downstream tasks. These findings highlight the importance of systematic robustness testing for understanding model behaviors. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_17311 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation Otmakhova, Yulia Truong, Hung Thinh Mahendra, Rahmad Zhai, Zenan Zhu, Rongxin Beck, Daniel Lau, Jey Han Computation and Language Artificial Intelligence We present FLUKE (Framework for LingUistically-driven and tasK-agnostic robustness Evaluation), a framework for assessing model robustness through systematic minimal variations of test data. FLUKE introduces controlled variations across linguistic levels -- from orthography to dialect and style -- and leverages large language models (LLMs) with human validation to generate modifications. We demonstrate FLUKE's utility by evaluating both fine-tuned models and LLMs across six diverse NLP tasks (four classification and two generation tasks), and reveal that (1) the impact of linguistic variations is highly task-dependent, with some tests being critical for certain tasks but irrelevant for others; (2) LLMs still exhibit significant brittleness to certain linguistic variations, with reasoning LLMs surprisingly showing less robustness on some tasks compared to base models, and scaling improving robustness only for surface-level modifications; (3) models are overall more brittle to natural, fluent modifications such as syntax or style changes (and especially to negation), compared to corruption-style tests such as letter flipping; (4) the ability of a model to use a linguistic feature in generation does not correlate to its robustness to this feature on downstream tasks. These findings highlight the importance of systematic robustness testing for understanding model behaviors. |
| title | FLUKE: A Linguistically-Driven and Task-Agnostic Framework for Robustness Evaluation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.17311 |