Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908684825657344 |
|---|---|
| author | Tran, Quoc-Huy Mehmood, Ahmed Ahmed, Muhammad Naufil, Muhammad Zafar, Anas Konin, Andrey Zia, M. Zeeshan |
| author_facet | Tran, Quoc-Huy Mehmood, Ahmed Ahmed, Muhammad Naufil, Muhammad Zafar, Anas Konin, Andrey Zia, M. Zeeshan |
| contents | This paper presents an unsupervised transformer-based framework for temporal activity segmentation which leverages not only frame-level cues but also segment-level cues. This is in contrast with previous methods which often rely on frame-level information only. Our approach begins with a frame-level prediction module which estimates framewise action classes via a transformer encoder. The frame-level prediction module is trained in an unsupervised manner via temporal optimal transport. To exploit segment-level information, we utilize a segment-level prediction module and a frame-to-segment alignment module. The former includes a transformer decoder for estimating video transcripts, while the latter matches frame-level features with segment-level features, yielding permutation-aware segmentation results. Moreover, inspired by temporal optimal transport, we introduce simple-yet-effective pseudo labels for unsupervised training of the above modules. Our experiments on four public datasets, i.e., 50 Salads, YouTube Instructions, Breakfast, and Desktop Assembly show that our approach achieves comparable or better performance than previous methods in unsupervised activity segmentation. Our code and dataset are available on our research website: https://retrocausal.ai/research/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_19478 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment Tran, Quoc-Huy Mehmood, Ahmed Ahmed, Muhammad Naufil, Muhammad Zafar, Anas Konin, Andrey Zia, M. Zeeshan Computer Vision and Pattern Recognition This paper presents an unsupervised transformer-based framework for temporal activity segmentation which leverages not only frame-level cues but also segment-level cues. This is in contrast with previous methods which often rely on frame-level information only. Our approach begins with a frame-level prediction module which estimates framewise action classes via a transformer encoder. The frame-level prediction module is trained in an unsupervised manner via temporal optimal transport. To exploit segment-level information, we utilize a segment-level prediction module and a frame-to-segment alignment module. The former includes a transformer decoder for estimating video transcripts, while the latter matches frame-level features with segment-level features, yielding permutation-aware segmentation results. Moreover, inspired by temporal optimal transport, we introduce simple-yet-effective pseudo labels for unsupervised training of the above modules. Our experiments on four public datasets, i.e., 50 Salads, YouTube Instructions, Breakfast, and Desktop Assembly show that our approach achieves comparable or better performance than previous methods in unsupervised activity segmentation. Our code and dataset are available on our research website: https://retrocausal.ai/research/. |
| title | Permutation-Aware Action Segmentation via Unsupervised Frame-to-Segment Alignment |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2305.19478 |