Emergence Without Scale: Inductive Resonance and Invariant Dynamics in Tiny Recurrent Networks
Fuente:
Zenodo
Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Recurso digital |
| Lingua: | inglese |
| Pubblicazione: |
Zenodo
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866902320137109504 |
|---|---|
| author | Giebelhaus, M. Axel |
| author_facet | Giebelhaus, M. Axel |
| contents | <p>Emergence in neural sequence models is typically framed as a scaling phenomenon, diagnosed by loss discontinuities or probe performance. We propose a mechanistic alternative: emergence is the acquisition of a hidden state transition operator F : ht → ht+1 that aligns with the latent dynamical laws of the data generator. We investigate this in minimal settings using single-layer tanh RNNs and GRUs (H ∈ [2, 32]) on synthetic worlds with known latent structure (nonlinear oscillators) and matched spectral nulls.</p> <p>While both architectures achieve predictive gains once capacity matches latent dimensionality, their internal mechanisms differ fundamentally. Lyapunov spectrum analysis reveals that trained GRUs develop a "high-Q resonant" regime: the leading Lyapunov exponent approaches neutral stability (λ1 ≈ 0) to preserve flow along the limit cycle, while transverse exponents are strongly contracting (λ>1 ≪ 0). In contrast, tanh RNNs behave as forced damped oscillators, where even the leading exponent is significantly contracting (λ1 < 0). Coasting experiments confirm this: when input forcing is removed, GRUs exhibit high-Q resonance, retaining phase and amplitude significantly longer than the strongly dissipative RNNs.</p> <p>Crucially, we find that the GRU's resonant structure is an architectural prior, not purely a learned behavior. When trained on null worlds (noise), the GRU converges to the same near-neutral spectrum and produces coherent oscillatory hallucinations, whereas the RNN correctly collapses to a stable sink. We conclude that emergence in gated networks is driven by inductive resonance: the gating mechanism enforces a quasi-conservative dynamical prior that leads to efficient learning when matched with oscillatory worlds, but coherent hallucination when mismatched with noise.</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_17952632 |
| institution | Zenodo |
| language | eng |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Emergence Without Scale: Inductive Resonance and Invariant Dynamics in Tiny Recurrent Networks Giebelhaus, M. Axel recurrent neural networks GRU emergence Lyapunov exponents dynamical systems inductive bias hallucination detection neural network dynamics machine learning <p>Emergence in neural sequence models is typically framed as a scaling phenomenon, diagnosed by loss discontinuities or probe performance. We propose a mechanistic alternative: emergence is the acquisition of a hidden state transition operator F : ht → ht+1 that aligns with the latent dynamical laws of the data generator. We investigate this in minimal settings using single-layer tanh RNNs and GRUs (H ∈ [2, 32]) on synthetic worlds with known latent structure (nonlinear oscillators) and matched spectral nulls.</p> <p>While both architectures achieve predictive gains once capacity matches latent dimensionality, their internal mechanisms differ fundamentally. Lyapunov spectrum analysis reveals that trained GRUs develop a "high-Q resonant" regime: the leading Lyapunov exponent approaches neutral stability (λ1 ≈ 0) to preserve flow along the limit cycle, while transverse exponents are strongly contracting (λ>1 ≪ 0). In contrast, tanh RNNs behave as forced damped oscillators, where even the leading exponent is significantly contracting (λ1 < 0). Coasting experiments confirm this: when input forcing is removed, GRUs exhibit high-Q resonance, retaining phase and amplitude significantly longer than the strongly dissipative RNNs.</p> <p>Crucially, we find that the GRU's resonant structure is an architectural prior, not purely a learned behavior. When trained on null worlds (noise), the GRU converges to the same near-neutral spectrum and produces coherent oscillatory hallucinations, whereas the RNN correctly collapses to a stable sink. We conclude that emergence in gated networks is driven by inductive resonance: the gating mechanism enforces a quasi-conservative dynamical prior that leads to efficient learning when matched with oscillatory worlds, but coherent hallucination when mismatched with noise.</p> |
| title | Emergence Without Scale: Inductive Resonance and Invariant Dynamics in Tiny Recurrent Networks |
| topic | recurrent neural networks GRU emergence Lyapunov exponents dynamical systems inductive bias hallucination detection neural network dynamics machine learning |
| url | https://doi.org/10.5281/zenodo.17952632 |