Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.09184 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916770177089536 |
|---|---|
| author | Finke, Lennart Sreedhara, Chandan Dooms, Thomas Allen, Mat Zhang, Emerald Rodriguez, Juan Diego Nabeshima, Noa Marshall, Thomas Braun, Dan |
| author_facet | Finke, Lennart Sreedhara, Chandan Dooms, Thomas Allen, Mat Zhang, Emerald Rodriguez, Juan Diego Nabeshima, Noa Marshall, Thomas Braun, Dan |
| contents | We present SimpleStories, a large synthetic story dataset in simple language, consisting of 2 million samples each in English and Japanese. Through parameterizing prompts at multiple levels of abstraction, we achieve control over story characteristics at scale, inducing syntactic and semantic diversity. Ablations on a newly trained model suite show improved sample efficiency and model interpretability compared to the TinyStories dataset. We open-source all constituent parts of model creation, hoping to enable novel ways to study the end-to-end training process. As a byproduct, we move the frontier regarding the fewest-parameter language model that outputs grammatical natural language. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_09184 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Parameterized Synthetic Text Generation with SimpleStories Finke, Lennart Sreedhara, Chandan Dooms, Thomas Allen, Mat Zhang, Emerald Rodriguez, Juan Diego Nabeshima, Noa Marshall, Thomas Braun, Dan Computation and Language Artificial Intelligence We present SimpleStories, a large synthetic story dataset in simple language, consisting of 2 million samples each in English and Japanese. Through parameterizing prompts at multiple levels of abstraction, we achieve control over story characteristics at scale, inducing syntactic and semantic diversity. Ablations on a newly trained model suite show improved sample efficiency and model interpretability compared to the TinyStories dataset. We open-source all constituent parts of model creation, hoping to enable novel ways to study the end-to-end training process. As a byproduct, we move the frontier regarding the fewest-parameter language model that outputs grammatical natural language. |
| title | Parameterized Synthetic Text Generation with SimpleStories |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.09184 |