Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2307.10867 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913896805171200 |
|---|---|
| author | Singh, Ashish Singh, Ashutosh Agarwal, Prateek Huang, Zixuan Singh, Arpita Yu, Tong Kim, Sungchul Bursztyn, Victor Ahmed, Nesreen K. Mathur, Puneet Learned-Miller, Erik Dernoncourt, Franck Rossi, Ryan A. |
| author_facet | Singh, Ashish Singh, Ashutosh Agarwal, Prateek Huang, Zixuan Singh, Arpita Yu, Tong Kim, Sungchul Bursztyn, Victor Ahmed, Nesreen K. Mathur, Puneet Learned-Miller, Erik Dernoncourt, Franck Rossi, Ryan A. |
| contents | Captions are crucial for understanding scientific visualizations and documents. Existing captioning methods for scientific figures rely on figure-caption pairs extracted from documents for training, many of which fall short with respect to metrics like helpfulness, explainability, and visual-descriptiveness [15] leading to generated captions being misaligned with reader preferences. To enable the generation of high-quality figure captions, we introduce FigCaps-HF a new framework for figure-caption generation that can incorporate domain expert feedback in generating captions optimized for reader preferences. Our framework comprises of 1) an automatic method for evaluating quality of figure-caption pairs, 2) a novel reinforcement learning with human feedback (RLHF) method to optimize a generative figure-to-caption model for reader preferences. We demonstrate the effectiveness of our simple learning framework by improving performance over standard fine-tuning across different types of models. In particular, when using BLIP as the base model, our RLHF framework achieves a mean gain of 35.7%, 16.9%, and 9% in ROUGE, BLEU, and Meteor, respectively. Finally, we release a large-scale benchmark dataset with human feedback on figure-caption pairs to enable further evaluation and development of RLHF techniques for this problem. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_10867 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback Singh, Ashish Singh, Ashutosh Agarwal, Prateek Huang, Zixuan Singh, Arpita Yu, Tong Kim, Sungchul Bursztyn, Victor Ahmed, Nesreen K. Mathur, Puneet Learned-Miller, Erik Dernoncourt, Franck Rossi, Ryan A. Computation and Language Computer Vision and Pattern Recognition Machine Learning Captions are crucial for understanding scientific visualizations and documents. Existing captioning methods for scientific figures rely on figure-caption pairs extracted from documents for training, many of which fall short with respect to metrics like helpfulness, explainability, and visual-descriptiveness [15] leading to generated captions being misaligned with reader preferences. To enable the generation of high-quality figure captions, we introduce FigCaps-HF a new framework for figure-caption generation that can incorporate domain expert feedback in generating captions optimized for reader preferences. Our framework comprises of 1) an automatic method for evaluating quality of figure-caption pairs, 2) a novel reinforcement learning with human feedback (RLHF) method to optimize a generative figure-to-caption model for reader preferences. We demonstrate the effectiveness of our simple learning framework by improving performance over standard fine-tuning across different types of models. In particular, when using BLIP as the base model, our RLHF framework achieves a mean gain of 35.7%, 16.9%, and 9% in ROUGE, BLEU, and Meteor, respectively. Finally, we release a large-scale benchmark dataset with human feedback on figure-caption pairs to enable further evaluation and development of RLHF techniques for this problem. |
| title | FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback |
| topic | Computation and Language Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2307.10867 |