Guardado en:
| Autor principal: | |
|---|---|
| Formato: | Recurso digital |
| Lenguaje: | |
| Publicado: |
Zenodo
2026
|
| Acceso en línea: | https://doi.org/10.5281/zenodo.18626464 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
Tabla de Contenidos:
- <p><span lang="EN-GB">We introduce <strong>Entropy Bandwidth Y-Axis (EBY)</strong>, a scalar diagnostic metric designed to quantify the behavioural quality of autoregressive language models during training. While conventional metrics such as cross-entropy loss and perplexity measure predictive accuracy, they provide no direct signal about the shape of the model’s output distribution—specifically, the balance between diversity and confidence.</span></p> <p><span lang="EN-GB">EBY captures this balance by combining normalized Shannon entropy with top-</span><span lang="EN-GB"></span><span lang="EN-GB"><span> </span>probability mass (sharpness), yielding a bounded signal in </span><span lang="EN-GB"></span><span lang="EN-GB">. Crucially, both degenerate regimes—deterministic collapse and incoherent randomness—map to low EBY values, while only balanced, human-like predictive behavior yields high EBY.</span></p> <p><span lang="EN-GB">We formalize EBY, analyse its asymptotic behaviour, introduce positional variants (EBY-Head and EBY-Tail) for temporal degradation detection, and describe a trend-based monitoring system suitable for online training. Empirical results from training a 381M-parameter Turkish language model demonstrate that EBY detects distributional collapse several hundred optimization steps before perplexity degradation becomes visible, enabling early corrective intervention. EBY thus provides a complementary behavioural axis for model diagnostics, orthogonal to loss-based metrics.</span></p>