Training Noise Token Pruning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912274083479552 |
|---|---|
| author | Rao, Mingxing Jiang, Bohan Moyer, Daniel |
| author_facet | Rao, Mingxing Jiang, Bohan Moyer, Daniel |
| contents | In the present work we present Training Noise Token (TNT) Pruning for vision transformers. Our method relaxes the discrete token dropping condition to continuous additive noise, providing smooth optimization in training, while retaining discrete dropping computational gains in deployment settings. We provide theoretical connections to Rate-Distortion literature, and empirical evaluations on the ImageNet dataset using ViT and DeiT architectures demonstrating TNT's advantages over previous pruning methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_18092 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Training Noise Token Pruning Rao, Mingxing Jiang, Bohan Moyer, Daniel Computer Vision and Pattern Recognition In the present work we present Training Noise Token (TNT) Pruning for vision transformers. Our method relaxes the discrete token dropping condition to continuous additive noise, providing smooth optimization in training, while retaining discrete dropping computational gains in deployment settings. We provide theoretical connections to Rate-Distortion literature, and empirical evaluations on the ImageNet dataset using ViT and DeiT architectures demonstrating TNT's advantages over previous pruning methods. |
| title | Training Noise Token Pruning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.18092 |