Extracting Moore Machines from Transformers using Queries and Counterexamples
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911179985649664 |
|---|---|
| author | Adriaensen, Rik Maene, Jaron |
| author_facet | Adriaensen, Rik Maene, Jaron |
| contents | Fuelled by the popularity of the transformer architecture in deep learning, several works have investigated what formal languages a transformer can learn from data. Nonetheless, existing results remain hard to compare due to methodological differences. To address this, we construct finite state automata as high-level abstractions of transformers trained on regular languages using queries and counterexamples. Concretely, we extract Moore machines, as many training tasks used in literature can be mapped onto them. We demonstrate the usefulness of this approach by studying positive-only learning and the sequence accuracy measure in detail. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_06045 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Extracting Moore Machines from Transformers using Queries and Counterexamples Adriaensen, Rik Maene, Jaron Machine Learning Artificial Intelligence Fuelled by the popularity of the transformer architecture in deep learning, several works have investigated what formal languages a transformer can learn from data. Nonetheless, existing results remain hard to compare due to methodological differences. To address this, we construct finite state automata as high-level abstractions of transformers trained on regular languages using queries and counterexamples. Concretely, we extract Moore machines, as many training tasks used in literature can be mapped onto them. We demonstrate the usefulness of this approach by studying positive-only learning and the sequence accuracy measure in detail. |
| title | Extracting Moore Machines from Transformers using Queries and Counterexamples |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.06045 |