Enregistré dans:
| Auteur principal: | |
|---|---|
| Format: | Recurso digital |
| Langue: | |
| Publié: |
Zenodo
2026
|
| Accès en ligne: | https://doi.org/10.5281/zenodo.18626464 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866901746206375936 |
|---|---|
| author | Özbay, Oğuzhan |
| author_facet | Özbay, Oğuzhan |
| contents | <p><span lang="EN-GB">We introduce <strong>Entropy Bandwidth Y-Axis (EBY)</strong>, a scalar diagnostic metric designed to quantify the behavioural quality of autoregressive language models during training. While conventional metrics such as cross-entropy loss and perplexity measure predictive accuracy, they provide no direct signal about the shape of the model’s output distribution—specifically, the balance between diversity and confidence.</span></p> <p><span lang="EN-GB">EBY captures this balance by combining normalized Shannon entropy with top-</span><span lang="EN-GB"></span><span lang="EN-GB"><span> </span>probability mass (sharpness), yielding a bounded signal in </span><span lang="EN-GB"></span><span lang="EN-GB">. Crucially, both degenerate regimes—deterministic collapse and incoherent randomness—map to low EBY values, while only balanced, human-like predictive behavior yields high EBY.</span></p> <p><span lang="EN-GB">We formalize EBY, analyse its asymptotic behaviour, introduce positional variants (EBY-Head and EBY-Tail) for temporal degradation detection, and describe a trend-based monitoring system suitable for online training. Empirical results from training a 381M-parameter Turkish language model demonstrate that EBY detects distributional collapse several hundred optimization steps before perplexity degradation becomes visible, enabling early corrective intervention. EBY thus provides a complementary behavioural axis for model diagnostics, orthogonal to loss-based metrics.</span></p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_18626464 |
| institution | Zenodo |
| language | |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Entropy Bandwidth Y-Axis (EBY) Özbay, Oğuzhan <p><span lang="EN-GB">We introduce <strong>Entropy Bandwidth Y-Axis (EBY)</strong>, a scalar diagnostic metric designed to quantify the behavioural quality of autoregressive language models during training. While conventional metrics such as cross-entropy loss and perplexity measure predictive accuracy, they provide no direct signal about the shape of the model’s output distribution—specifically, the balance between diversity and confidence.</span></p> <p><span lang="EN-GB">EBY captures this balance by combining normalized Shannon entropy with top-</span><span lang="EN-GB"></span><span lang="EN-GB"><span> </span>probability mass (sharpness), yielding a bounded signal in </span><span lang="EN-GB"></span><span lang="EN-GB">. Crucially, both degenerate regimes—deterministic collapse and incoherent randomness—map to low EBY values, while only balanced, human-like predictive behavior yields high EBY.</span></p> <p><span lang="EN-GB">We formalize EBY, analyse its asymptotic behaviour, introduce positional variants (EBY-Head and EBY-Tail) for temporal degradation detection, and describe a trend-based monitoring system suitable for online training. Empirical results from training a 381M-parameter Turkish language model demonstrate that EBY detects distributional collapse several hundred optimization steps before perplexity degradation becomes visible, enabling early corrective intervention. EBY thus provides a complementary behavioural axis for model diagnostics, orthogonal to loss-based metrics.</span></p> |
| title | Entropy Bandwidth Y-Axis (EBY) |
| url | https://doi.org/10.5281/zenodo.18626464 |