Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2021
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2106.15332 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915785200369664 |
|---|---|
| author | Qiao, Yixuan Chen, Hao Wang, Jun Zhao, Shanshan Chen, Yihao Ye, Xianbin Li, Ziliang Qi, Xianbiao Gao, Peng Xie, Guotong |
| author_facet | Qiao, Yixuan Chen, Hao Wang, Jun Zhao, Shanshan Chen, Yihao Ye, Xianbin Li, Ziliang Qi, Xianbiao Gao, Peng Xie, Guotong |
| contents | TextVQA requires models to read and reason about text in images to answer questions about them. Specifically, models need to incorporate a new modality of text present in the images and reason over it to answer TextVQA questions. In this challenge, we use generative model T5 for TextVQA task. Based on pre-trained checkpoint T5-3B from HuggingFace repository, two other pre-training tasks including masked language modeling(MLM) and relative position prediction(RPP) are designed to better align object feature and scene text. In the stage of pre-training, encoder is dedicate to handle the fusion among multiple modalities: question text, object text labels, scene text labels, object visual features, scene visual features. After that decoder generates the text sequence step-by-step, cross entropy loss is required by default. We use a large-scale scene text dataset in pre-training and then fine-tune the T5-3B with the TextVQA dataset only. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2106_15332 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model Qiao, Yixuan Chen, Hao Wang, Jun Zhao, Shanshan Chen, Yihao Ye, Xianbin Li, Ziliang Qi, Xianbiao Gao, Peng Xie, Guotong Computer Vision and Pattern Recognition TextVQA requires models to read and reason about text in images to answer questions about them. Specifically, models need to incorporate a new modality of text present in the images and reason over it to answer TextVQA questions. In this challenge, we use generative model T5 for TextVQA task. Based on pre-trained checkpoint T5-3B from HuggingFace repository, two other pre-training tasks including masked language modeling(MLM) and relative position prediction(RPP) are designed to better align object feature and scene text. In the stage of pre-training, encoder is dedicate to handle the fusion among multiple modalities: question text, object text labels, scene text labels, object visual features, scene visual features. After that decoder generates the text sequence step-by-step, cross entropy loss is required by default. We use a large-scale scene text dataset in pre-training and then fine-tune the T5-3B with the TextVQA dataset only. |
| title | Winner Team Mia at TextVQA Challenge 2021: Vision-and-Language Representation Learning with Pre-trained Sequence-to-Sequence Model |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2106.15332 |