A One-Layer Decoder-Only Transformer is a Two-Layer RNN: With an Application to Certified Robustness
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910460373106688 |
|---|---|
| author | Zhang, Yuhao Albarghouthi, Aws D'Antoni, Loris |
| author_facet | Zhang, Yuhao Albarghouthi, Aws D'Antoni, Loris |
| contents | This paper reveals a key insight that a one-layer decoder-only Transformer is equivalent to a two-layer Recurrent Neural Network (RNN). Building on this insight, we propose ARC-Tran, a novel approach for verifying the robustness of decoder-only Transformers against arbitrary perturbation spaces. Compared to ARC-Tran, current robustness verification techniques are limited either to specific and length-preserving perturbations like word substitutions or to recursive models like LSTMs. ARC-Tran addresses these limitations by meticulously managing position encoding to prevent mismatches and by utilizing our key insight to achieve precise and scalable verification. Our evaluation shows that ARC-Tran (1) trains models more robust to arbitrary perturbation spaces than those produced by existing techniques and (2) shows high certification accuracy of the resulting models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17361 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A One-Layer Decoder-Only Transformer is a Two-Layer RNN: With an Application to Certified Robustness Zhang, Yuhao Albarghouthi, Aws D'Antoni, Loris Computation and Language This paper reveals a key insight that a one-layer decoder-only Transformer is equivalent to a two-layer Recurrent Neural Network (RNN). Building on this insight, we propose ARC-Tran, a novel approach for verifying the robustness of decoder-only Transformers against arbitrary perturbation spaces. Compared to ARC-Tran, current robustness verification techniques are limited either to specific and length-preserving perturbations like word substitutions or to recursive models like LSTMs. ARC-Tran addresses these limitations by meticulously managing position encoding to prevent mismatches and by utilizing our key insight to achieve precise and scalable verification. Our evaluation shows that ARC-Tran (1) trains models more robust to arbitrary perturbation spaces than those produced by existing techniques and (2) shows high certification accuracy of the resulting models. |
| title | A One-Layer Decoder-Only Transformer is a Two-Layer RNN: With an Application to Certified Robustness |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.17361 |