Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exiting
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913357889536000 |
|---|---|
| author | Hamed, Omar Bakkali, Souhail Moens, Marie-Francine Blaschko, Matthew Van Landeghem, Jordy |
| author_facet | Hamed, Omar Bakkali, Souhail Moens, Marie-Francine Blaschko, Matthew Van Landeghem, Jordy |
| contents | This work addresses the need for a balanced approach between performance and efficiency in scalable production environments for visually-rich document understanding (VDU) tasks. Currently, there is a reliance on large document foundation models that offer advanced capabilities but come with a heavy computational burden. In this paper, we propose a multimodal early exit (EE) model design that incorporates various training strategies, exit layer types and placements. Our goal is to achieve a Pareto-optimal balance between predictive performance and efficiency for multimodal document image classification. Through a comprehensive set of experiments, we compare our approach with traditional exit policies and showcase an improved performance-efficiency trade-off. Our multimodal EE design preserves the model's predictive capabilities, enhancing both speed and latency. This is achieved through a reduction of over 20% in latency, while fully retaining the baseline accuracy. This research represents the first exploration of multimodal EE design within the VDU community, highlighting as well the effectiveness of calibration in improving confidence scores for exiting at different layers. Overall, our findings contribute to practical VDU applications by enhancing both performance and efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_12705 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exiting Hamed, Omar Bakkali, Souhail Moens, Marie-Francine Blaschko, Matthew Van Landeghem, Jordy Computer Vision and Pattern Recognition Computation and Language Machine Learning This work addresses the need for a balanced approach between performance and efficiency in scalable production environments for visually-rich document understanding (VDU) tasks. Currently, there is a reliance on large document foundation models that offer advanced capabilities but come with a heavy computational burden. In this paper, we propose a multimodal early exit (EE) model design that incorporates various training strategies, exit layer types and placements. Our goal is to achieve a Pareto-optimal balance between predictive performance and efficiency for multimodal document image classification. Through a comprehensive set of experiments, we compare our approach with traditional exit policies and showcase an improved performance-efficiency trade-off. Our multimodal EE design preserves the model's predictive capabilities, enhancing both speed and latency. This is achieved through a reduction of over 20% in latency, while fully retaining the baseline accuracy. This research represents the first exploration of multimodal EE design within the VDU community, highlighting as well the effectiveness of calibration in improving confidence scores for exiting at different layers. Overall, our findings contribute to practical VDU applications by enhancing both performance and efficiency. |
| title | Multimodal Adaptive Inference for Document Image Classification with Anytime Early Exiting |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2405.12705 |