How BERT Speaks Shakespearean English? Evaluating Historical Bias in Contextual Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929237150138368 |
|---|---|
| author | Cuscito, Miriam Ferrara, Alfio Ruskov, Martin |
| author_facet | Cuscito, Miriam Ferrara, Alfio Ruskov, Martin |
| contents | In this paper, we explore the idea of analysing the historical bias of contextual language models based on BERT by measuring their adequacy with respect to Early Modern (EME) and Modern (ME) English. In our preliminary experiments, we perform fill-in-the-blank tests with 60 masked sentences (20 EME-specific, 20 ME-specific and 20 generic) and three different models (i.e., BERT Base, MacBERTh, English HLM). We then rate the model predictions according to a 5-point bipolar scale between the two language varieties and derive a weighted score to measure the adequacy of each model to EME and ME varieties of English. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05034 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | How BERT Speaks Shakespearean English? Evaluating Historical Bias in Contextual Language Models Cuscito, Miriam Ferrara, Alfio Ruskov, Martin Computation and Language Computers and Society I.2.7; J.5 In this paper, we explore the idea of analysing the historical bias of contextual language models based on BERT by measuring their adequacy with respect to Early Modern (EME) and Modern (ME) English. In our preliminary experiments, we perform fill-in-the-blank tests with 60 masked sentences (20 EME-specific, 20 ME-specific and 20 generic) and three different models (i.e., BERT Base, MacBERTh, English HLM). We then rate the model predictions according to a 5-point bipolar scale between the two language varieties and derive a weighted score to measure the adequacy of each model to EME and ME varieties of English. |
| title | How BERT Speaks Shakespearean English? Evaluating Historical Bias in Contextual Language Models |
| topic | Computation and Language Computers and Society I.2.7; J.5 |
| url | https://arxiv.org/abs/2402.05034 |