Training Trajectory Structure: Regime Detection, Cross-Seed Convergence, and Speculative Weight Prediction
Fuente:
Zenodo
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Recurso digital |
| Veröffentlicht: |
Zenodo
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866901180297248768 |
|---|---|
| author | McEntire, Jeremy |
| author_facet | McEntire, Jeremy |
| contents | Training trajectories alternate between stable basins where gradients are predictable and chaotic transitions where they are not. We show these regimes are detectable via activation fingerprints, synchronized across independent seeds, and exploitable for speculative weight prediction. Consolidates and supersedes the Leap+Verify (DOI: 10.5281/zenodo.18828410) and Ensemble Collapse (DOI: 10.5281/zenodo.18828412) papers into a unified treatment. Linear prediction achieves 60-90% strict acceptance at 7B scale in stable regimes. Momentum-based prediction fails catastrophically at all scales (100-10,000x loss inflation). Phase boundaries synchronize across 5 independent seeds to within +/-50 steps. Final validation loss CV is 0.41% (124M), 2.43% (1.5B), 1.53% (7B). |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_19432379 |
| institution | Zenodo |
| language | |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Training Trajectory Structure: Regime Detection, Cross-Seed Convergence, and Speculative Weight Prediction McEntire, Jeremy training trajectories regime detection speculative prediction ensemble convergence activation fingerprinting Training trajectories alternate between stable basins where gradients are predictable and chaotic transitions where they are not. We show these regimes are detectable via activation fingerprints, synchronized across independent seeds, and exploitable for speculative weight prediction. Consolidates and supersedes the Leap+Verify (DOI: 10.5281/zenodo.18828410) and Ensemble Collapse (DOI: 10.5281/zenodo.18828412) papers into a unified treatment. Linear prediction achieves 60-90% strict acceptance at 7B scale in stable regimes. Momentum-based prediction fails catastrophically at all scales (100-10,000x loss inflation). Phase boundaries synchronize across 5 independent seeds to within +/-50 steps. Final validation loss CV is 0.41% (124M), 2.43% (1.5B), 1.53% (7B). |
| title | Training Trajectory Structure: Regime Detection, Cross-Seed Convergence, and Speculative Weight Prediction |
| topic | training trajectories regime detection speculative prediction ensemble convergence activation fingerprinting |
| url | https://doi.org/10.5281/zenodo.19432379 |