Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.15846 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917438220664832 |
|---|---|
| author | Gao, Xinyu Wang, Shaonan Ding, Nai |
| author_facet | Gao, Xinyu Wang, Shaonan Ding, Nai |
| contents | Decoder-only large language models achieve strong broad performance but are brittle to minor grammatical perturbations, undermining reliability for downstream reasoning. However, directly injecting explicit syntactic structure into an existing checkpoint can interfere with its pretrained competence. We introduce a checkpoint-compatible gated tree cross-attention (GTCA) branch that reads precomputed constituency chunk memory while leaving backbone architecture unchanged. Our design uses a token update mask and staged training to control the scope and timing of structural updates. Across benchmarks and Transformer backbones, GTCA strengthens syntactic robustness beyond continued-training baselines without compromising Multiple-Choice QA performance or commonsense reasoning, providing a practical checkpoint-compatible route to more syntax-robust decoder-only LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_15846 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs Gao, Xinyu Wang, Shaonan Ding, Nai Computation and Language Decoder-only large language models achieve strong broad performance but are brittle to minor grammatical perturbations, undermining reliability for downstream reasoning. However, directly injecting explicit syntactic structure into an existing checkpoint can interfere with its pretrained competence. We introduce a checkpoint-compatible gated tree cross-attention (GTCA) branch that reads precomputed constituency chunk memory while leaving backbone architecture unchanged. Our design uses a token update mask and staged training to control the scope and timing of structural updates. Across benchmarks and Transformer backbones, GTCA strengthens syntactic robustness beyond continued-training baselines without compromising Multiple-Choice QA performance or commonsense reasoning, providing a practical checkpoint-compatible route to more syntax-robust decoder-only LLMs. |
| title | Gated Tree Cross-Attention for Checkpoint-Compatible Syntax Injection in Decoder-Only LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.15846 |