Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918461599383552 |
|---|---|
| author | Zhu, Yihua Liu, Qianying Wang, Jiaxin Cheng, Fei Liu, Chaoran Aizawa, Akiko Kurohashi, Sadao Shimodaira, Hidetoshi |
| author_facet | Zhu, Yihua Liu, Qianying Wang, Jiaxin Cheng, Fei Liu, Chaoran Aizawa, Akiko Kurohashi, Sadao Shimodaira, Hidetoshi |
| contents | Autoregressive LLMs perform well on relational tasks that require linking entities via relational words (e.g., father/son, friend), but it is unclear whether they learn the logical semantics of such relations (e.g., symmetry and inversion logic) and, if so, whether reversal-type failures arise from missing relational semantics or left-to-right order bias. We propose a controlled Knowledge Graph-based synthetic framework that generates text from symmetric/inverse triples, train GPT-style autoregressive models from scratch, and evaluate memorization, logical inference, and in-context generalization to unseen entities to address these questions. We find a sharp phase transition in which relational semantics emerge with sufficient logic-bearing supervision, even in shallow (2-3 layer) models, and that successful generalization aligns with stable intermediate-layer signals. Finally, order-matched forward/reverse tests and a diffusion baseline indicate that reversal failures are primarily driven by autoregressive order bias rather than deficient inversion semantics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_02931 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMs Zhu, Yihua Liu, Qianying Wang, Jiaxin Cheng, Fei Liu, Chaoran Aizawa, Akiko Kurohashi, Sadao Shimodaira, Hidetoshi Computation and Language Autoregressive LLMs perform well on relational tasks that require linking entities via relational words (e.g., father/son, friend), but it is unclear whether they learn the logical semantics of such relations (e.g., symmetry and inversion logic) and, if so, whether reversal-type failures arise from missing relational semantics or left-to-right order bias. We propose a controlled Knowledge Graph-based synthetic framework that generates text from symmetric/inverse triples, train GPT-style autoregressive models from scratch, and evaluate memorization, logical inference, and in-context generalization to unseen entities to address these questions. We find a sharp phase transition in which relational semantics emerge with sufficient logic-bearing supervision, even in shallow (2-3 layer) models, and that successful generalization aligns with stable intermediate-layer signals. Finally, order-matched forward/reverse tests and a diffusion baseline indicate that reversal failures are primarily driven by autoregressive order bias rather than deficient inversion semantics. |
| title | Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.02931 |