Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.07532 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910525748674560 |
|---|---|
| author | Zou, Xueyan Li, Linjie Wang, Jianfeng Yang, Jianwei Ding, Mingyu Wei, Junyi Yang, Zhengyuan Li, Feng Zhang, Hao Liu, Shilong Aravinthan, Arul Lee, Yong Jae Wang, Lijuan |
| author_facet | Zou, Xueyan Li, Linjie Wang, Jianfeng Yang, Jianwei Ding, Mingyu Wei, Junyi Yang, Zhengyuan Li, Feng Zhang, Hao Liu, Shilong Aravinthan, Arul Lee, Yong Jae Wang, Lijuan |
| contents | Foundation models possess strong capabilities in reasoning and memorizing across modalities. To further unleash the power of foundation models, we present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity. As shown in the teaser figure, a lightweight transformer interface without tuning any foundation model weights is enough for segmentation, grounding, and retrieval in an interleaved manner. The proposed interface has the following favorable attributes: (1) Generalizable. It applies to various tasks spanning retrieval, segmentation, etc., under the same architecture and weights. (2) Interleavable. With the benefit of multi-task multi-modal training, the proposed interface creates an interleaved shared embedding space. (3) Extendable. The proposed interface is adaptive to new tasks, and new models. In light of the interleaved embedding space, we introduce FIND-Bench, which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval. We are the first work aligning foundations models' embeddings for interleave understanding. Meanwhile, our approach achieves state-of-the-art performance on FIND-Bench and competitive performance on standard retrieval and segmentation settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_07532 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Interfacing Foundation Models' Embeddings Zou, Xueyan Li, Linjie Wang, Jianfeng Yang, Jianwei Ding, Mingyu Wei, Junyi Yang, Zhengyuan Li, Feng Zhang, Hao Liu, Shilong Aravinthan, Arul Lee, Yong Jae Wang, Lijuan Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Foundation models possess strong capabilities in reasoning and memorizing across modalities. To further unleash the power of foundation models, we present FIND, a generalized interface for aligning foundation models' embeddings with unified image and dataset-level understanding spanning modality and granularity. As shown in the teaser figure, a lightweight transformer interface without tuning any foundation model weights is enough for segmentation, grounding, and retrieval in an interleaved manner. The proposed interface has the following favorable attributes: (1) Generalizable. It applies to various tasks spanning retrieval, segmentation, etc., under the same architecture and weights. (2) Interleavable. With the benefit of multi-task multi-modal training, the proposed interface creates an interleaved shared embedding space. (3) Extendable. The proposed interface is adaptive to new tasks, and new models. In light of the interleaved embedding space, we introduce FIND-Bench, which introduces new training and evaluation annotations to the COCO dataset for interleaved segmentation and retrieval. We are the first work aligning foundations models' embeddings for interleave understanding. Meanwhile, our approach achieves state-of-the-art performance on FIND-Bench and competitive performance on standard retrieval and segmentation settings. |
| title | Interfacing Foundation Models' Embeddings |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2312.07532 |