Social Good or Scientific Curiosity? Uncovering the Research Framing Behind NLP Artefacts
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916989425942528 |
|---|---|
| author | Chamoun, Eric Ousidhoum, Nedjma Schlichtkrull, Michael Vlachos, Andreas |
| author_facet | Chamoun, Eric Ousidhoum, Nedjma Schlichtkrull, Michael Vlachos, Andreas |
| contents | Clarifying the research framing of NLP artefacts (e.g., models, datasets, etc.) is crucial to aligning research with practical applications. Recent studies manually analyzed NLP research across domains, showing that few papers explicitly identify key stakeholders, intended uses, or appropriate contexts. In this work, we propose to automate this analysis, developing a three-component system that infers research framings by first extracting key elements (means, ends, stakeholders), then linking them through interpretable rules and contextual reasoning. We evaluate our approach on two domains: automated fact-checking using an existing dataset, and hate speech detection for which we annotate a new dataset-achieving consistent improvements over strong LLM baselines. Finally, we apply our system to recent automated fact-checking papers and uncover three notable trends: a rise in vague or underspecified research goals, increased emphasis on scientific exploration over application, and a shift toward supporting human fact-checkers rather than pursuing full automation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_18677 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Social Good or Scientific Curiosity? Uncovering the Research Framing Behind NLP Artefacts Chamoun, Eric Ousidhoum, Nedjma Schlichtkrull, Michael Vlachos, Andreas Computation and Language Clarifying the research framing of NLP artefacts (e.g., models, datasets, etc.) is crucial to aligning research with practical applications. Recent studies manually analyzed NLP research across domains, showing that few papers explicitly identify key stakeholders, intended uses, or appropriate contexts. In this work, we propose to automate this analysis, developing a three-component system that infers research framings by first extracting key elements (means, ends, stakeholders), then linking them through interpretable rules and contextual reasoning. We evaluate our approach on two domains: automated fact-checking using an existing dataset, and hate speech detection for which we annotate a new dataset-achieving consistent improvements over strong LLM baselines. Finally, we apply our system to recent automated fact-checking papers and uncover three notable trends: a rise in vague or underspecified research goals, increased emphasis on scientific exploration over application, and a shift toward supporting human fact-checkers rather than pursuing full automation. |
| title | Social Good or Scientific Curiosity? Uncovering the Research Framing Behind NLP Artefacts |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.18677 |