Saved in:
| Hovedforfatter: | |
|---|---|
| Format: | Recurso digital |
| Sprog: | |
| Udgivet: |
Zenodo
2026
|
| Online adgang: | https://doi.org/10.5281/zenodo.18930213 |
| Tags: |
Tilføj Tag
Ingen Tags, Vær først til at tagge denne postø!
|
Indholdsfortegnelse:
- <p>Catastrophic forgetting arises when gradient updates for a new task overwrite parameter directions critical to a previously learned task. We argue that the information field tensor Gamma_info -- a curvature object derived from the entropy functional of the model's predictive distribution [Li 2026] -- provides a geometry-informed signal for continual learning: directions in the approximate null space of Gamma_info are information-neutral and potentially safe to update.</p> <p>We instantiate this view through an audit-gated gradient projection family. Rather than claiming exact full-parameter null-space recovery, we use Gamma cross-batch reproducibility audits: each parameter's Gamma_info sub-block is estimated on two disjoint batch halves, and only parameters whose near-null eigenspaces align across both halves (Criterion A > 0.5, passing in >= 2 independent subsets) enter gradient projection. A matched random-direction control (same support indices and subspace rank) isolates whether the audit-identified direction -- not merely the projection operation -- is the source of any forgetting benefit.</p> <p>In a cross-domain continual learning experiment (GPT-2, WikiText-2 -> Biomedical Medical QA), the audit gates 41-42 of 42 candidate parameters across 5 random seeds, demonstrating robust null-space structure throughout GPT-2 layers h.6-h.11. Audit-gated null projection (gamma_along) significantly reduces forgetting versus unconstrained fine-tuning (+331 +/- 30 vs. +414 +/- 45, Delta = -83, p < 0.05, 5 seeds), while preserving Task B perplexity (9.55 vs. 9.54 for free). The direction signal is directionally present: gamma_along < gamma_random (Delta = -38), supporting the geometric claim that audit-identified null directions -- not merely projection -- reduce forgetting.</p>