DiffAugment: Diffusion based Long-Tailed Visual Relationship Recognition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913868957089792 |
|---|---|
| author | Gupta, Parul Nguyen, Tuan Dhall, Abhinav Hayat, Munawar Le, Trung Do, Thanh-Toan |
| author_facet | Gupta, Parul Nguyen, Tuan Dhall, Abhinav Hayat, Munawar Le, Trung Do, Thanh-Toan |
| contents | The task of Visual Relationship Recognition (VRR) aims to identify relationships between two interacting objects in an image and is particularly challenging due to the widely-spread and highly imbalanced distribution of <subject, relation, object> triplets. To overcome the resultant performance bias in existing VRR approaches, we introduce DiffAugment -- a method which first augments the tail classes in the linguistic space by making use of WordNet and then utilizes the generative prowess of Diffusion Models to expand the visual space for minority classes. We propose a novel hardness-aware component in diffusion which is based upon the hardness of each <S,R,O> triplet and demonstrate the effectiveness of hardness-aware diffusion in generating visual embeddings for the tail classes. We also propose a novel subject and object based seeding strategy for diffusion sampling which improves the discriminative capability of the generated visual embeddings. Extensive experimentation on the GQA-LT dataset shows favorable gains in the subject/object and relation average per-class accuracy using Diffusion augmented samples. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_01387 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DiffAugment: Diffusion based Long-Tailed Visual Relationship Recognition Gupta, Parul Nguyen, Tuan Dhall, Abhinav Hayat, Munawar Le, Trung Do, Thanh-Toan Computer Vision and Pattern Recognition The task of Visual Relationship Recognition (VRR) aims to identify relationships between two interacting objects in an image and is particularly challenging due to the widely-spread and highly imbalanced distribution of <subject, relation, object> triplets. To overcome the resultant performance bias in existing VRR approaches, we introduce DiffAugment -- a method which first augments the tail classes in the linguistic space by making use of WordNet and then utilizes the generative prowess of Diffusion Models to expand the visual space for minority classes. We propose a novel hardness-aware component in diffusion which is based upon the hardness of each <S,R,O> triplet and demonstrate the effectiveness of hardness-aware diffusion in generating visual embeddings for the tail classes. We also propose a novel subject and object based seeding strategy for diffusion sampling which improves the discriminative capability of the generated visual embeddings. Extensive experimentation on the GQA-LT dataset shows favorable gains in the subject/object and relation average per-class accuracy using Diffusion augmented samples. |
| title | DiffAugment: Diffusion based Long-Tailed Visual Relationship Recognition |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2401.01387 |