Enriching language models with graph-based context information to better understand textual data
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916884421541888 |
|---|---|
| author | Roethel, Albert Ganzha, Maria Wróblewska, Anna |
| author_facet | Roethel, Albert Ganzha, Maria Wróblewska, Anna |
| contents | A considerable number of texts encountered daily are somehow connected with each other. For example, Wikipedia articles refer to other articles via hyperlinks, scientific papers relate to others via citations or (co)authors, while tweets relate via users that follow each other or reshare content. Hence, a graph-like structure can represent existing connections and be seen as capturing the "context" of the texts. The question thus arises if extracting and integrating such context information into a language model might help facilitate a better automated understanding of the text. In this study, we experimentally demonstrate that incorporating graph-based contextualization into BERT model enhances its performance on an example of a classification task. Specifically, on Pubmed dataset, we observed a reduction in error from 8.51% to 7.96%, while increasing the number of parameters just by 1.6%.
Our source code: https://github.com/tryptofanik/gc-bert |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_11070 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Enriching language models with graph-based context information to better understand textual data Roethel, Albert Ganzha, Maria Wróblewska, Anna Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing A considerable number of texts encountered daily are somehow connected with each other. For example, Wikipedia articles refer to other articles via hyperlinks, scientific papers relate to others via citations or (co)authors, while tweets relate via users that follow each other or reshare content. Hence, a graph-like structure can represent existing connections and be seen as capturing the "context" of the texts. The question thus arises if extracting and integrating such context information into a language model might help facilitate a better automated understanding of the text. In this study, we experimentally demonstrate that incorporating graph-based contextualization into BERT model enhances its performance on an example of a classification task. Specifically, on Pubmed dataset, we observed a reduction in error from 8.51% to 7.96%, while increasing the number of parameters just by 1.6%. Our source code: https://github.com/tryptofanik/gc-bert |
| title | Enriching language models with graph-based context information to better understand textual data |
| topic | Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2305.11070 |