Friends Across Time: Multi-Scale Action Segmentation Transformer for Surgical Phase Recognition
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916101058723840 |
|---|---|
| author | Zhang, Bokai Meng, Jiayuan Cheng, Bin Biskup, Dean Petculescu, Svetlana Chapman, Angela |
| author_facet | Zhang, Bokai Meng, Jiayuan Cheng, Bin Biskup, Dean Petculescu, Svetlana Chapman, Angela |
| contents | Automatic surgical phase recognition is a core technology for modern operating rooms and online surgical video assessment platforms. Current state-of-the-art methods use both spatial and temporal information to tackle the surgical phase recognition task. Building on this idea, we propose the Multi-Scale Action Segmentation Transformer (MS-AST) for offline surgical phase recognition and the Multi-Scale Action Segmentation Causal Transformer (MS-ASCT) for online surgical phase recognition. We use ResNet50 or EfficientNetV2-M for spatial feature extraction. Our MS-AST and MS-ASCT can model temporal information at different scales with multi-scale temporal self-attention and multi-scale temporal cross-attention, which enhances the capture of temporal relationships between frames and segments. We demonstrate that our method can achieve 95.26% and 96.15% accuracy on the Cholec80 dataset for online and offline surgical phase recognition, respectively, which achieves new state-of-the-art results. Our method can also achieve state-of-the-art results on non-medical datasets in the video action segmentation domain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_11644 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Friends Across Time: Multi-Scale Action Segmentation Transformer for Surgical Phase Recognition Zhang, Bokai Meng, Jiayuan Cheng, Bin Biskup, Dean Petculescu, Svetlana Chapman, Angela Computer Vision and Pattern Recognition Robotics Automatic surgical phase recognition is a core technology for modern operating rooms and online surgical video assessment platforms. Current state-of-the-art methods use both spatial and temporal information to tackle the surgical phase recognition task. Building on this idea, we propose the Multi-Scale Action Segmentation Transformer (MS-AST) for offline surgical phase recognition and the Multi-Scale Action Segmentation Causal Transformer (MS-ASCT) for online surgical phase recognition. We use ResNet50 or EfficientNetV2-M for spatial feature extraction. Our MS-AST and MS-ASCT can model temporal information at different scales with multi-scale temporal self-attention and multi-scale temporal cross-attention, which enhances the capture of temporal relationships between frames and segments. We demonstrate that our method can achieve 95.26% and 96.15% accuracy on the Cholec80 dataset for online and offline surgical phase recognition, respectively, which achieves new state-of-the-art results. Our method can also achieve state-of-the-art results on non-medical datasets in the video action segmentation domain. |
| title | Friends Across Time: Multi-Scale Action Segmentation Transformer for Surgical Phase Recognition |
| topic | Computer Vision and Pattern Recognition Robotics |
| url | https://arxiv.org/abs/2401.11644 |