Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915287928930304 |
|---|---|
| author | Venkatesh, Danush Kumar Funke, Isabel Pfeiffer, Micha Kolbinger, Fiona Schmeiser, Hanna Maria Weitz, Juergen Distler, Marius Speidel, Stefanie |
| author_facet | Venkatesh, Danush Kumar Funke, Isabel Pfeiffer, Micha Kolbinger, Fiona Schmeiser, Hanna Maria Weitz, Juergen Distler, Marius Speidel, Stefanie |
| contents | Computer-assisted interventions can improve intra-operative guidance, particularly through deep learning methods that harness the spatiotemporal information in surgical videos. However, the severe data imbalance often found in surgical video datasets hinders the development of high-performing models. In this work, we aim to overcome the data imbalance by synthesizing surgical videos. We propose a unique two-stage, text-conditioned diffusion-based method to generate high-fidelity surgical videos for under-represented classes. Our approach conditions the generation process on text prompts and decouples spatial and temporal modeling by utilizing a 2D latent diffusion model to capture spatial content and then integrating temporal attention layers to ensure temporal consistency. Furthermore, we introduce a rejection sampling strategy to select the most suitable synthetic samples, effectively augmenting existing datasets to address class imbalance. We evaluate our method on two downstream tasks-surgical action recognition and intra-operative event prediction-demonstrating that incorporating synthetic videos from our approach substantially enhances model performance. We open-source our implementation at https://gitlab.com/nct_tso_public/surgvgen. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_09858 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models Venkatesh, Danush Kumar Funke, Isabel Pfeiffer, Micha Kolbinger, Fiona Schmeiser, Hanna Maria Weitz, Juergen Distler, Marius Speidel, Stefanie Computer Vision and Pattern Recognition Computer-assisted interventions can improve intra-operative guidance, particularly through deep learning methods that harness the spatiotemporal information in surgical videos. However, the severe data imbalance often found in surgical video datasets hinders the development of high-performing models. In this work, we aim to overcome the data imbalance by synthesizing surgical videos. We propose a unique two-stage, text-conditioned diffusion-based method to generate high-fidelity surgical videos for under-represented classes. Our approach conditions the generation process on text prompts and decouples spatial and temporal modeling by utilizing a 2D latent diffusion model to capture spatial content and then integrating temporal attention layers to ensure temporal consistency. Furthermore, we introduce a rejection sampling strategy to select the most suitable synthetic samples, effectively augmenting existing datasets to address class imbalance. We evaluate our method on two downstream tasks-surgical action recognition and intra-operative event prediction-demonstrating that incorporating synthetic videos from our approach substantially enhances model performance. We open-source our implementation at https://gitlab.com/nct_tso_public/surgvgen. |
| title | Mission Balance: Generating Under-represented Class Samples using Video Diffusion Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.09858 |