Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929632484261888 |
|---|---|
| author | Mąka, Paweł Semerci, Yusuf Can Scholtes, Jan Spanakis, Gerasimos |
| author_facet | Mąka, Paweł Semerci, Yusuf Can Scholtes, Jan Spanakis, Gerasimos |
| contents | In this paper, we investigate the role of attention heads in Context-aware Machine Translation models for pronoun disambiguation in the English-to-German and English-to-French language directions. We analyze their influence by both observing and modifying the attention scores corresponding to the plausible relations that could impact a pronoun prediction. Our findings reveal that while some heads do attend the relations of interest, not all of them influence the models' ability to disambiguate pronouns. We show that certain heads are underutilized by the models, suggesting that model performance could be improved if only the heads would attend one of the relations more strongly. Furthermore, we fine-tune the most promising heads and observe the increase in pronoun disambiguation accuracy of up to 5 percentage points which demonstrates that the improvements in performance can be solidified into the models' parameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_11187 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models Mąka, Paweł Semerci, Yusuf Can Scholtes, Jan Spanakis, Gerasimos Computation and Language Artificial Intelligence Machine Learning In this paper, we investigate the role of attention heads in Context-aware Machine Translation models for pronoun disambiguation in the English-to-German and English-to-French language directions. We analyze their influence by both observing and modifying the attention scores corresponding to the plausible relations that could impact a pronoun prediction. Our findings reveal that while some heads do attend the relations of interest, not all of them influence the models' ability to disambiguate pronouns. We show that certain heads are underutilized by the models, suggesting that model performance could be improved if only the heads would attend one of the relations more strongly. Furthermore, we fine-tune the most promising heads and observe the increase in pronoun disambiguation accuracy of up to 5 percentage points which demonstrates that the improvements in performance can be solidified into the models' parameters. |
| title | Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2412.11187 |