Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.17247 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911889194221568 |
|---|---|
| author | Bordes, Florian Pang, Richard Yuanzhe Ajay, Anurag Li, Alexander C. Bardes, Adrien Petryk, Suzanne Mañas, Oscar Lin, Zhiqiu Mahmoud, Anas Jayaraman, Bargav Ibrahim, Mark Hall, Melissa Xiong, Yunyang Lebensold, Jonathan Ross, Candace Jayakumar, Srihari Guo, Chuan Bouchacourt, Diane Al-Tahan, Haider Padthe, Karthik Sharma, Vasu Xu, Hu Tan, Xiaoqing Ellen Richards, Megan Lavoie, Samuel Astolfi, Pietro Hemmat, Reyhane Askari Chen, Jun Tirumala, Kushal Assouel, Rim Moayeri, Mazda Talattof, Arjang Chaudhuri, Kamalika Liu, Zechun Chen, Xilun Garrido, Quentin Ullrich, Karen Agrawal, Aishwarya Saenko, Kate Celikyilmaz, Asli Chandra, Vikas |
| author_facet | Bordes, Florian Pang, Richard Yuanzhe Ajay, Anurag Li, Alexander C. Bardes, Adrien Petryk, Suzanne Mañas, Oscar Lin, Zhiqiu Mahmoud, Anas Jayaraman, Bargav Ibrahim, Mark Hall, Melissa Xiong, Yunyang Lebensold, Jonathan Ross, Candace Jayakumar, Srihari Guo, Chuan Bouchacourt, Diane Al-Tahan, Haider Padthe, Karthik Sharma, Vasu Xu, Hu Tan, Xiaoqing Ellen Richards, Megan Lavoie, Samuel Astolfi, Pietro Hemmat, Reyhane Askari Chen, Jun Tirumala, Kushal Assouel, Rim Moayeri, Mazda Talattof, Arjang Chaudhuri, Kamalika Liu, Zechun Chen, Xilun Garrido, Quentin Ullrich, Karen Agrawal, Aishwarya Saenko, Kate Celikyilmaz, Asli Chandra, Vikas |
| contents | Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision to language, we present this introduction to VLMs which we hope will help anyone who would like to enter the field. First, we introduce what VLMs are, how they work, and how to train them. Then, we present and discuss approaches to evaluate VLMs. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17247 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Introduction to Vision-Language Modeling Bordes, Florian Pang, Richard Yuanzhe Ajay, Anurag Li, Alexander C. Bardes, Adrien Petryk, Suzanne Mañas, Oscar Lin, Zhiqiu Mahmoud, Anas Jayaraman, Bargav Ibrahim, Mark Hall, Melissa Xiong, Yunyang Lebensold, Jonathan Ross, Candace Jayakumar, Srihari Guo, Chuan Bouchacourt, Diane Al-Tahan, Haider Padthe, Karthik Sharma, Vasu Xu, Hu Tan, Xiaoqing Ellen Richards, Megan Lavoie, Samuel Astolfi, Pietro Hemmat, Reyhane Askari Chen, Jun Tirumala, Kushal Assouel, Rim Moayeri, Mazda Talattof, Arjang Chaudhuri, Kamalika Liu, Zechun Chen, Xilun Garrido, Quentin Ullrich, Karen Agrawal, Aishwarya Saenko, Kate Celikyilmaz, Asli Chandra, Vikas Machine Learning Following the recent popularity of Large Language Models (LLMs), several attempts have been made to extend them to the visual domain. From having a visual assistant that could guide us through unfamiliar environments to generative models that produce images using only a high-level text description, the vision-language model (VLM) applications will significantly impact our relationship with technology. However, there are many challenges that need to be addressed to improve the reliability of those models. While language is discrete, vision evolves in a much higher dimensional space in which concepts cannot always be easily discretized. To better understand the mechanics behind mapping vision to language, we present this introduction to VLMs which we hope will help anyone who would like to enter the field. First, we introduce what VLMs are, how they work, and how to train them. Then, we present and discuss approaches to evaluate VLMs. Although this work primarily focuses on mapping images to language, we also discuss extending VLMs to videos. |
| title | An Introduction to Vision-Language Modeling |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2405.17247 |