SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916159416172544 |
|---|---|
| author | Zhang, Yuxuan Song, Yiren Liu, Jiaming Wang, Rui Yu, Jinpeng Tang, Hao Li, Huaxia Tang, Xu Hu, Yao Pan, Han Jing, Zhongliang |
| author_facet | Zhang, Yuxuan Song, Yiren Liu, Jiaming Wang, Rui Yu, Jinpeng Tang, Hao Li, Huaxia Tang, Xu Hu, Yao Pan, Han Jing, Zhongliang |
| contents | Recent advancements in subject-driven image generation have led to zero-shot generation, yet precise selection and focus on crucial subject representations remain challenging. Addressing this, we introduce the SSR-Encoder, a novel architecture designed for selectively capturing any subject from single or multiple reference images. It responds to various query modalities including text and masks, without necessitating test-time fine-tuning. The SSR-Encoder combines a Token-to-Patch Aligner that aligns query inputs with image patches and a Detail-Preserving Subject Encoder for extracting and preserving fine features of the subjects, thereby generating subject embeddings. These embeddings, used in conjunction with original text embeddings, condition the generation process. Characterized by its model generalizability and efficiency, the SSR-Encoder adapts to a range of custom models and control modules. Enhanced by the Embedding Consistency Regularization Loss for improved training, our extensive experiments demonstrate its effectiveness in versatile and high-quality image generation, indicating its broad applicability. Project page: https://ssr-encoder.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_16272 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation Zhang, Yuxuan Song, Yiren Liu, Jiaming Wang, Rui Yu, Jinpeng Tang, Hao Li, Huaxia Tang, Xu Hu, Yao Pan, Han Jing, Zhongliang Computer Vision and Pattern Recognition Recent advancements in subject-driven image generation have led to zero-shot generation, yet precise selection and focus on crucial subject representations remain challenging. Addressing this, we introduce the SSR-Encoder, a novel architecture designed for selectively capturing any subject from single or multiple reference images. It responds to various query modalities including text and masks, without necessitating test-time fine-tuning. The SSR-Encoder combines a Token-to-Patch Aligner that aligns query inputs with image patches and a Detail-Preserving Subject Encoder for extracting and preserving fine features of the subjects, thereby generating subject embeddings. These embeddings, used in conjunction with original text embeddings, condition the generation process. Characterized by its model generalizability and efficiency, the SSR-Encoder adapts to a range of custom models and control modules. Enhanced by the Embedding Consistency Regularization Loss for improved training, our extensive experiments demonstrate its effectiveness in versatile and high-quality image generation, indicating its broad applicability. Project page: https://ssr-encoder.github.io |
| title | SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.16272 |