LatentKeypointGAN: Controlling Images via Latent Keypoints
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2021
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914970699038720 |
|---|---|
| author | He, Xingzhe Wandt, Bastian Rhodin, Helge |
| author_facet | He, Xingzhe Wandt, Bastian Rhodin, Helge |
| contents | Generative adversarial networks (GANs) have attained photo-realistic quality in image generation. However, how to best control the image content remains an open challenge. We introduce LatentKeypointGAN, a two-stage GAN which is trained end-to-end on the classical GAN objective with internal conditioning on a set of space keypoints. These keypoints have associated appearance embeddings that respectively control the position and style of the generated objects and their parts. A major difficulty that we address with suitable network architectures and training schemes is disentangling the image into spatial and appearance factors without domain knowledge and supervision signals. We demonstrate that LatentKeypointGAN provides an interpretable latent space that can be used to re-arrange the generated images by re-positioning and exchanging keypoint embeddings, such as generating portraits by combining the eyes, nose, and mouth from different images. In addition, the explicit generation of keypoints and matching images enables a new, GAN-based method for unsupervised keypoint detection. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2103_15812 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | LatentKeypointGAN: Controlling Images via Latent Keypoints He, Xingzhe Wandt, Bastian Rhodin, Helge Computer Vision and Pattern Recognition Generative adversarial networks (GANs) have attained photo-realistic quality in image generation. However, how to best control the image content remains an open challenge. We introduce LatentKeypointGAN, a two-stage GAN which is trained end-to-end on the classical GAN objective with internal conditioning on a set of space keypoints. These keypoints have associated appearance embeddings that respectively control the position and style of the generated objects and their parts. A major difficulty that we address with suitable network architectures and training schemes is disentangling the image into spatial and appearance factors without domain knowledge and supervision signals. We demonstrate that LatentKeypointGAN provides an interpretable latent space that can be used to re-arrange the generated images by re-positioning and exchanging keypoint embeddings, such as generating portraits by combining the eyes, nose, and mouth from different images. In addition, the explicit generation of keypoints and matching images enables a new, GAN-based method for unsupervised keypoint detection. |
| title | LatentKeypointGAN: Controlling Images via Latent Keypoints |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2103.15812 |