What do Transformers Know about Government?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913324205080576 |
|---|---|
| author | Hou, Jue Katinskaia, Anisia Kotilainen, Lari Trangcasanchai, Sathianpong Vu, Anh-Duc Yangarber, Roman |
| author_facet | Hou, Jue Katinskaia, Anisia Kotilainen, Lari Trangcasanchai, Sathianpong Vu, Anh-Duc Yangarber, Roman |
| contents | This paper investigates what insights about linguistic features and what knowledge about the structure of natural language can be obtained from the encodings in transformer language models.In particular, we explore how BERT encodes the government relation between constituents in a sentence. We use several probing classifiers, and data from two morphologically rich languages. Our experiments show that information about government is encoded across all transformer layers, but predominantly in the early layers of the model. We find that, for both languages, a small number of attention heads encode enough information about the government relations to enable us to train a classifier capable of discovering new, previously unknown types of government, never seen in the training data. Currently, data is lacking for the research community working on grammatical constructions, and government in particular. We release the Government Bank -- a dataset defining the government relations for thousands of lemmas in the languages in our experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_14270 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | What do Transformers Know about Government? Hou, Jue Katinskaia, Anisia Kotilainen, Lari Trangcasanchai, Sathianpong Vu, Anh-Duc Yangarber, Roman Computation and Language Machine Learning This paper investigates what insights about linguistic features and what knowledge about the structure of natural language can be obtained from the encodings in transformer language models.In particular, we explore how BERT encodes the government relation between constituents in a sentence. We use several probing classifiers, and data from two morphologically rich languages. Our experiments show that information about government is encoded across all transformer layers, but predominantly in the early layers of the model. We find that, for both languages, a small number of attention heads encode enough information about the government relations to enable us to train a classifier capable of discovering new, previously unknown types of government, never seen in the training data. Currently, data is lacking for the research community working on grammatical constructions, and government in particular. We release the Government Bank -- a dataset defining the government relations for thousands of lemmas in the languages in our experiments. |
| title | What do Transformers Know about Government? |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2404.14270 |