ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912093938122752 |
|---|---|
| author | Newman, Benjamin Lee, Yoonjoo Naik, Aakanksha Siangliulue, Pao Fok, Raymond Kim, Juho Weld, Daniel S. Chang, Joseph Chee Lo, Kyle |
| author_facet | Newman, Benjamin Lee, Yoonjoo Naik, Aakanksha Siangliulue, Pao Fok, Raymond Kim, Juho Weld, Daniel S. Chang, Joseph Chee Lo, Kyle |
| contents | When conducting literature reviews, scientists often create literature review tables - tables whose rows are publications and whose columns constitute a schema, a set of aspects used to compare and contrast the papers. Can we automatically generate these tables using language models (LMs)? In this work, we introduce a framework that leverages LMs to perform this task by decomposing it into separate schema and value generation steps. To enable experimentation, we address two main challenges: First, we overcome a lack of high-quality datasets to benchmark table generation by curating and releasing arxivDIGESTables, a new dataset of 2,228 literature review tables extracted from ArXiv papers that synthesize a total of 7,542 research papers. Second, to support scalable evaluation of model generations against human-authored reference tables, we develop DecontextEval, an automatic evaluation method that aligns elements of tables with the same underlying aspects despite differing surface forms. Given these tools, we evaluate LMs' abilities to reconstruct reference tables, finding this task benefits from additional context to ground the generation (e.g. table captions, in-text references). Finally, through a human evaluation study we find that even when LMs fail to fully reconstruct a reference table, their generated novel aspects can still be useful. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_22360 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models Newman, Benjamin Lee, Yoonjoo Naik, Aakanksha Siangliulue, Pao Fok, Raymond Kim, Juho Weld, Daniel S. Chang, Joseph Chee Lo, Kyle Computation and Language When conducting literature reviews, scientists often create literature review tables - tables whose rows are publications and whose columns constitute a schema, a set of aspects used to compare and contrast the papers. Can we automatically generate these tables using language models (LMs)? In this work, we introduce a framework that leverages LMs to perform this task by decomposing it into separate schema and value generation steps. To enable experimentation, we address two main challenges: First, we overcome a lack of high-quality datasets to benchmark table generation by curating and releasing arxivDIGESTables, a new dataset of 2,228 literature review tables extracted from ArXiv papers that synthesize a total of 7,542 research papers. Second, to support scalable evaluation of model generations against human-authored reference tables, we develop DecontextEval, an automatic evaluation method that aligns elements of tables with the same underlying aspects despite differing surface forms. Given these tools, we evaluate LMs' abilities to reconstruct reference tables, finding this task benefits from additional context to ground the generation (e.g. table captions, in-text references). Finally, through a human evaluation study we find that even when LMs fail to fully reconstruct a reference table, their generated novel aspects can still be useful. |
| title | ArxivDIGESTables: Synthesizing Scientific Literature into Tables using Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.22360 |