How PARTs assemble into wholes: Learning the relative composition of images
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915673693749248 |
|---|---|
| author | Ayoughi, Melika Abnar, Samira Huang, Chen Sandino, Chris Lala, Sayeri Dhekane, Eeshan Gunesh Busbridge, Dan Zhai, Shuangfei Thilak, Vimal Susskind, Josh Mettes, Pascal Groth, Paul Goh, Hanlin |
| author_facet | Ayoughi, Melika Abnar, Samira Huang, Chen Sandino, Chris Lala, Sayeri Dhekane, Eeshan Gunesh Busbridge, Dan Zhai, Shuangfei Thilak, Vimal Susskind, Josh Mettes, Pascal Groth, Paul Goh, Hanlin |
| contents | The composition of objects and their parts, along with object-object positional relationships, provides a rich source of information for representation learning. Hence, spatial-aware pretext tasks have been actively explored in self-supervised learning. Existing works commonly start from a grid structure, where the goal of the pretext task involves predicting the absolute position index of patches within a fixed grid. However, grid-based approaches fall short of capturing the fluid and continuous nature of real-world object compositions. We introduce PART, a self-supervised learning approach that leverages continuous relative transformations between off-grid patches to overcome these limitations. By modeling how parts relate to each other in a continuous space, PART learns the relative composition of images-an off-grid structural relative positioning that is less tied to absolute appearance and can remain coherent under variations such as partial visibility or stylistic changes. In tasks requiring precise spatial understanding such as object detection and time series prediction, PART outperforms grid-based methods like MAE and DropPos, while maintaining competitive performance on global classification tasks. By breaking free from grid constraints, PART opens up a new trajectory for universal self-supervised pretraining across diverse datatypes-from images to EEG signals-with potential in medical imaging, video, and audio. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_03682 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | How PARTs assemble into wholes: Learning the relative composition of images Ayoughi, Melika Abnar, Samira Huang, Chen Sandino, Chris Lala, Sayeri Dhekane, Eeshan Gunesh Busbridge, Dan Zhai, Shuangfei Thilak, Vimal Susskind, Josh Mettes, Pascal Groth, Paul Goh, Hanlin Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning The composition of objects and their parts, along with object-object positional relationships, provides a rich source of information for representation learning. Hence, spatial-aware pretext tasks have been actively explored in self-supervised learning. Existing works commonly start from a grid structure, where the goal of the pretext task involves predicting the absolute position index of patches within a fixed grid. However, grid-based approaches fall short of capturing the fluid and continuous nature of real-world object compositions. We introduce PART, a self-supervised learning approach that leverages continuous relative transformations between off-grid patches to overcome these limitations. By modeling how parts relate to each other in a continuous space, PART learns the relative composition of images-an off-grid structural relative positioning that is less tied to absolute appearance and can remain coherent under variations such as partial visibility or stylistic changes. In tasks requiring precise spatial understanding such as object detection and time series prediction, PART outperforms grid-based methods like MAE and DropPos, while maintaining competitive performance on global classification tasks. By breaking free from grid constraints, PART opens up a new trajectory for universal self-supervised pretraining across diverse datatypes-from images to EEG signals-with potential in medical imaging, video, and audio. |
| title | How PARTs assemble into wholes: Learning the relative composition of images |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2506.03682 |