MLRegTest: A Benchmark for the Machine Learning of Regular Languages
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929479834664960 |
|---|---|
| author | van der Poel, Sam Lambert, Dakotah Kostyszyn, Kalina Gao, Tiantian Verma, Rahul Andersen, Derek Chau, Joanne Peterson, Emily Clair, Cody St. Fodor, Paul Shibata, Chihiro Heinz, Jeffrey |
| author_facet | van der Poel, Sam Lambert, Dakotah Kostyszyn, Kalina Gao, Tiantian Verma, Rahul Andersen, Derek Chau, Joanne Peterson, Emily Clair, Cody St. Fodor, Paul Shibata, Chihiro Heinz, Jeffrey |
| contents | Synthetic datasets constructed from formal languages allow fine-grained examination of the learning and generalization capabilities of machine learning systems for sequence classification. This article presents a new benchmark for machine learning systems on sequence classification called MLRegTest, which contains training, development, and test sets from 1,800 regular languages. Different kinds of formal languages represent different kinds of long-distance dependencies, and correctly identifying long-distance dependencies in sequences is a known challenge for ML systems to generalize successfully. MLRegTest organizes its languages according to their logical complexity (monadic second order, first order, propositional, or monomial expressions) and the kind of logical literals (string, tier-string, subsequence, or combinations thereof). The logical complexity and choice of literal provides a systematic way to understand different kinds of long-distance dependencies in regular languages, and therefore to understand the capacities of different ML systems to learn such long-distance dependencies. Finally, the performance of different neural networks (simple RNN, LSTM, GRU, transformer) on MLRegTest is examined. The main conclusion is that performance depends significantly on the kind of test set, the class of language, and the neural network architecture. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_07687 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | MLRegTest: A Benchmark for the Machine Learning of Regular Languages van der Poel, Sam Lambert, Dakotah Kostyszyn, Kalina Gao, Tiantian Verma, Rahul Andersen, Derek Chau, Joanne Peterson, Emily Clair, Cody St. Fodor, Paul Shibata, Chihiro Heinz, Jeffrey Machine Learning Computation and Language Formal Languages and Automata Theory Synthetic datasets constructed from formal languages allow fine-grained examination of the learning and generalization capabilities of machine learning systems for sequence classification. This article presents a new benchmark for machine learning systems on sequence classification called MLRegTest, which contains training, development, and test sets from 1,800 regular languages. Different kinds of formal languages represent different kinds of long-distance dependencies, and correctly identifying long-distance dependencies in sequences is a known challenge for ML systems to generalize successfully. MLRegTest organizes its languages according to their logical complexity (monadic second order, first order, propositional, or monomial expressions) and the kind of logical literals (string, tier-string, subsequence, or combinations thereof). The logical complexity and choice of literal provides a systematic way to understand different kinds of long-distance dependencies in regular languages, and therefore to understand the capacities of different ML systems to learn such long-distance dependencies. Finally, the performance of different neural networks (simple RNN, LSTM, GRU, transformer) on MLRegTest is examined. The main conclusion is that performance depends significantly on the kind of test set, the class of language, and the neural network architecture. |
| title | MLRegTest: A Benchmark for the Machine Learning of Regular Languages |
| topic | Machine Learning Computation and Language Formal Languages and Automata Theory |
| url | https://arxiv.org/abs/2304.07687 |