Lumos : Empowering Multimodal LLMs with Scene Text Recognition
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929368977113088 |
|---|---|
| author | Shenoy, Ashish Lu, Yichao Jayakumar, Srihari Chatterjee, Debojeet Moslehpour, Mohsen Chuang, Pierce Harpale, Abhay Bhardwaj, Vikas Xu, Di Zhao, Shicong Zhao, Longfang Ramchandani, Ankit Dong, Xin Luna Kumar, Anuj |
| author_facet | Shenoy, Ashish Lu, Yichao Jayakumar, Srihari Chatterjee, Debojeet Moslehpour, Mohsen Chuang, Pierce Harpale, Abhay Bhardwaj, Vikas Xu, Di Zhao, Shicong Zhao, Longfang Ramchandani, Ankit Dong, Xin Luna Kumar, Anuj |
| contents | We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_08017 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Lumos : Empowering Multimodal LLMs with Scene Text Recognition Shenoy, Ashish Lu, Yichao Jayakumar, Srihari Chatterjee, Debojeet Moslehpour, Mohsen Chuang, Pierce Harpale, Abhay Bhardwaj, Vikas Xu, Di Zhao, Shicong Zhao, Longfang Ramchandani, Ankit Dong, Xin Luna Kumar, Anuj Computer Vision and Pattern Recognition Computation and Language Machine Learning We introduce Lumos, the first end-to-end multimodal question-answering system with text understanding capabilities. At the core of Lumos is a Scene Text Recognition (STR) component that extracts text from first person point-of-view images, the output of which is used to augment input to a Multimodal Large Language Model (MM-LLM). While building Lumos, we encountered numerous challenges related to STR quality, overall latency, and model inference. In this paper, we delve into those challenges, and discuss the system architecture, design choices, and modeling techniques employed to overcome these obstacles. We also provide a comprehensive evaluation for each component, showcasing high quality and efficiency. |
| title | Lumos : Empowering Multimodal LLMs with Scene Text Recognition |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2402.08017 |