Linguistic Profiling of a Neural Language Model
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2020
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913241895010304 |
|---|---|
| author | Miaschi, Alessio Brunato, Dominique Dell'Orletta, Felice Venturi, Giulia |
| author_facet | Miaschi, Alessio Brunato, Dominique Dell'Orletta, Felice Venturi, Giulia |
| contents | In this paper we investigate the linguistic knowledge learned by a Neural Language Model (NLM) before and after a fine-tuning process and how this knowledge affects its predictions during several classification problems. We use a wide set of probing tasks, each of which corresponds to a distinct sentence-level feature extracted from different levels of linguistic annotation. We show that BERT is able to encode a wide range of linguistic characteristics, but it tends to lose this information when trained on specific downstream tasks. We also find that BERT's capacity to encode different kind of linguistic properties has a positive influence on its predictions: the more it stores readable linguistic information of a sentence, the higher will be its capacity of predicting the expected label assigned to that sentence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2010_01869 |
| institution | arXiv |
| publishDate | 2020 |
| record_format | arxiv |
| spellingShingle | Linguistic Profiling of a Neural Language Model Miaschi, Alessio Brunato, Dominique Dell'Orletta, Felice Venturi, Giulia Computation and Language Artificial Intelligence In this paper we investigate the linguistic knowledge learned by a Neural Language Model (NLM) before and after a fine-tuning process and how this knowledge affects its predictions during several classification problems. We use a wide set of probing tasks, each of which corresponds to a distinct sentence-level feature extracted from different levels of linguistic annotation. We show that BERT is able to encode a wide range of linguistic characteristics, but it tends to lose this information when trained on specific downstream tasks. We also find that BERT's capacity to encode different kind of linguistic properties has a positive influence on its predictions: the more it stores readable linguistic information of a sentence, the higher will be its capacity of predicting the expected label assigned to that sentence. |
| title | Linguistic Profiling of a Neural Language Model |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2010.01869 |