Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Model
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916677802786816 |
|---|---|
| author | Wang, Shijian Song, Linxin Zhang, Jieyu Shimizu, Ryotaro Jin, Jiarui Luo, Ao Lu, Yuan Yao, Li Chen, Cunjian McAuley, Julian Zhang, Wentao Wu, Hanqian |
| author_facet | Wang, Shijian Song, Linxin Zhang, Jieyu Shimizu, Ryotaro Jin, Jiarui Luo, Ao Lu, Yuan Yao, Li Chen, Cunjian McAuley, Julian Zhang, Wentao Wu, Hanqian |
| contents | Current multimodal language model (MLM) training approaches overlook the influence of instruction templates. Previous research deals with this problem by leveraging hand-crafted or model-generated templates, failing to investigate the scaling effect of instruction templates on MLM training. In this work, we propose a programmatic instruction template generator capable of producing over 15K unique instruction templates by filling randomly sampled positional synonyms into weighted sampled meta templates, enabling us to comprehensively explore MLM's performance across various template scales in the training process. Our investigation into scaling instruction templates for MLM training demonstrates that MLM capabilities do not consistently improve with increasing template scale. Instead, optimal performance is achieved at a medium template scale. Models trained with data augmented at the optimal template scale achieve performance gains of up to 10% over those trained on the original data and achieve the best overall performance compared with the similar-scale MLMs tuned on at most 75 times the scale of our augmented dataset. The code will be publicly available at https://github.com/shijian2001/TemplateScaling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08307 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Model Wang, Shijian Song, Linxin Zhang, Jieyu Shimizu, Ryotaro Jin, Jiarui Luo, Ao Lu, Yuan Yao, Li Chen, Cunjian McAuley, Julian Zhang, Wentao Wu, Hanqian Computer Vision and Pattern Recognition Current multimodal language model (MLM) training approaches overlook the influence of instruction templates. Previous research deals with this problem by leveraging hand-crafted or model-generated templates, failing to investigate the scaling effect of instruction templates on MLM training. In this work, we propose a programmatic instruction template generator capable of producing over 15K unique instruction templates by filling randomly sampled positional synonyms into weighted sampled meta templates, enabling us to comprehensively explore MLM's performance across various template scales in the training process. Our investigation into scaling instruction templates for MLM training demonstrates that MLM capabilities do not consistently improve with increasing template scale. Instead, optimal performance is achieved at a medium template scale. Models trained with data augmented at the optimal template scale achieve performance gains of up to 10% over those trained on the original data and achieve the best overall performance compared with the similar-scale MLMs tuned on at most 75 times the scale of our augmented dataset. The code will be publicly available at https://github.com/shijian2001/TemplateScaling. |
| title | Investigating the Scaling Effect of Instruction Templates for Training Multimodal Language Model |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.08307 |