Evaluation Revisited: A Taxonomy of Evaluation Concerns in Natural Language Processing
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914514931286016 |
|---|---|
| author | Dhar, Ruchira Søgaard, Anders |
| author_facet | Dhar, Ruchira Søgaard, Anders |
| contents | Recent advances in large language models (LLMs) have prompted a growing body of work that questions the methodology of prevailing evaluation practices. However, many such critiques have already been extensively debated in natural language processing (NLP): a field with a long history of methodological reflection on evaluation. We conduct a scoping review of research on evaluation concerns in NLP and develop a taxonomy, synthesizing recurring positions and trade-offs within each area. We also discuss practical implications of the taxonomy, including a structured checklist to support more deliberate evaluation design and interpretation. By situating contemporary debates within their historical context, this work provides a consolidated reference for reasoning about evaluation practices. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_25923 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Evaluation Revisited: A Taxonomy of Evaluation Concerns in Natural Language Processing Dhar, Ruchira Søgaard, Anders Computation and Language Recent advances in large language models (LLMs) have prompted a growing body of work that questions the methodology of prevailing evaluation practices. However, many such critiques have already been extensively debated in natural language processing (NLP): a field with a long history of methodological reflection on evaluation. We conduct a scoping review of research on evaluation concerns in NLP and develop a taxonomy, synthesizing recurring positions and trade-offs within each area. We also discuss practical implications of the taxonomy, including a structured checklist to support more deliberate evaluation design and interpretation. By situating contemporary debates within their historical context, this work provides a consolidated reference for reasoning about evaluation practices. |
| title | Evaluation Revisited: A Taxonomy of Evaluation Concerns in Natural Language Processing |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.25923 |