Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.25073 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915522432466944 |
|---|---|
| author | Sun, Simeng |
| author_facet | Sun, Simeng |
| contents | We study Transformers on the task \emph{program trace generation} (PTG), where models produce step-by-step execution traces for synthetic programs. Unlike existing algorithmic problems, PTG externalizes reasoning through long traces where each step is trivial. We train small Transformers with diverse modifications, including alternative position encodings, softmax replacements, hybrid model, and short convolutions. While these models achieve strong in-distribution accuracy, they exhibit systematic failures when generalizing to various factors (e.g., program length, trace steps), though some designs significantly improve generalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_25073 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | An empirical study on the limitation of Transformers in program trace generation Sun, Simeng Computation and Language We study Transformers on the task \emph{program trace generation} (PTG), where models produce step-by-step execution traces for synthetic programs. Unlike existing algorithmic problems, PTG externalizes reasoning through long traces where each step is trivial. We train small Transformers with diverse modifications, including alternative position encodings, softmax replacements, hybrid model, and short convolutions. While these models achieve strong in-distribution accuracy, they exhibit systematic failures when generalizing to various factors (e.g., program length, trace steps), though some designs significantly improve generalization. |
| title | An empirical study on the limitation of Transformers in program trace generation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.25073 |