State-of-the-art generalisation research in NLP: A taxonomy and review
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2022
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910294802956288 |
|---|---|
| author | Hupkes, Dieuwke Giulianelli, Mario Dankers, Verna Artetxe, Mikel Elazar, Yanai Pimentel, Tiago Christodoulopoulos, Christos Lasri, Karim Saphra, Naomi Sinclair, Arabella Ulmer, Dennis Schottmann, Florian Batsuren, Khuyagbaatar Sun, Kaiser Sinha, Koustuv Khalatbari, Leila Ryskina, Maria Frieske, Rita Cotterell, Ryan Jin, Zhijing |
| author_facet | Hupkes, Dieuwke Giulianelli, Mario Dankers, Verna Artetxe, Mikel Elazar, Yanai Pimentel, Tiago Christodoulopoulos, Christos Lasri, Karim Saphra, Naomi Sinclair, Arabella Ulmer, Dennis Schottmann, Florian Batsuren, Khuyagbaatar Sun, Kaiser Sinha, Koustuv Khalatbari, Leila Ryskina, Maria Frieske, Rita Cotterell, Ryan Jin, Zhijing |
| contents | The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what 'good generalisation' entails and how it should be evaluated is not well understood, nor are there any evaluation standards for generalisation. In this paper, we lay the groundwork to address both of these issues. We present a taxonomy for characterising and understanding generalisation research in NLP. Our taxonomy is based on an extensive literature review of generalisation research, and contains five axes along which studies can differ: their main motivation, the type of generalisation they investigate, the type of data shift they consider, the source of this data shift, and the locus of the shift within the modelling pipeline. We use our taxonomy to classify over 400 papers that test generalisation, for a total of more than 600 individual experiments. Considering the results of this review, we present an in-depth analysis that maps out the current state of generalisation research in NLP, and we make recommendations for which areas might deserve attention in the future. Along with this paper, we release a webpage where the results of our review can be dynamically explored, and which we intend to update as new NLP generalisation studies are published. With this work, we aim to take steps towards making state-of-the-art generalisation testing the new status quo in NLP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2210_03050 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | State-of-the-art generalisation research in NLP: A taxonomy and review Hupkes, Dieuwke Giulianelli, Mario Dankers, Verna Artetxe, Mikel Elazar, Yanai Pimentel, Tiago Christodoulopoulos, Christos Lasri, Karim Saphra, Naomi Sinclair, Arabella Ulmer, Dennis Schottmann, Florian Batsuren, Khuyagbaatar Sun, Kaiser Sinha, Koustuv Khalatbari, Leila Ryskina, Maria Frieske, Rita Cotterell, Ryan Jin, Zhijing Computation and Language Artificial Intelligence The ability to generalise well is one of the primary desiderata of natural language processing (NLP). Yet, what 'good generalisation' entails and how it should be evaluated is not well understood, nor are there any evaluation standards for generalisation. In this paper, we lay the groundwork to address both of these issues. We present a taxonomy for characterising and understanding generalisation research in NLP. Our taxonomy is based on an extensive literature review of generalisation research, and contains five axes along which studies can differ: their main motivation, the type of generalisation they investigate, the type of data shift they consider, the source of this data shift, and the locus of the shift within the modelling pipeline. We use our taxonomy to classify over 400 papers that test generalisation, for a total of more than 600 individual experiments. Considering the results of this review, we present an in-depth analysis that maps out the current state of generalisation research in NLP, and we make recommendations for which areas might deserve attention in the future. Along with this paper, we release a webpage where the results of our review can be dynamically explored, and which we intend to update as new NLP generalisation studies are published. With this work, we aim to take steps towards making state-of-the-art generalisation testing the new status quo in NLP. |
| title | State-of-the-art generalisation research in NLP: A taxonomy and review |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2210.03050 |