| _version_ | 1866901820521054208 |
|---|---|
| author | Grandic, Sanjin Grandic, Sanjin |
| author_facet | Grandic, Sanjin Grandic, Sanjin |
| contents | <h2><strong>Entropy Bands for Scientific Reasoning in Probabilistic Transformer Models</strong></h2> <h3>Transformer-based LLMs rely on probabilistic decoding, and their stability depends on maintaining a workable entropy margin in the next-token distribution. Recent analyses have shown that when entropy collapses below a minimal threshold, safety masks and pruning mechanisms begin to conflict with the model’s logits. This mechanism explains both short-lived coherence in low-entropy settings and the subsequent onset of deviation, drift, or misinterpretation.</h3> <h3>This technical note extends that work by proposing <strong>discipline-specific entropy bands</strong> for stable scientific reasoning. Different fields place different demands on precision, abstraction, and flexibility. As a result, the entropy range required for coherent reasoning is not uniform across domains.</h3> <h3>-Mathematical reasoning requires a narrow, low-entropy band.</h3> <h3>-General scientific work requires a broader range.</h3> <h3>-Cross-disciplinary conceptual synthesis requires the upper part of the stable region.</h3> <h3>The paper defines three such bands and relates them to the minimal entropy threshold and the Grandić Tunnel Stability Region (GTSR). The analysis clarifies why LLM performance varies across tasks even when prompts are equally clear, and why pushing models into extreme low-entropy regimes can only be sustained briefly under strong external constraints.</h3> <h3><em><strong>This note completes a set of foundational tools for understanding the geometric and thermodynamic boundaries of probabilistic transformer models.</strong></em></h3> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_17730827 |
| institution | Zenodo |
| language | |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Entropy Bands for Scientific Reasoning in Probabilistic Transformer Models Grandic, Sanjin Grandic, Sanjin <h2><strong>Entropy Bands for Scientific Reasoning in Probabilistic Transformer Models</strong></h2> <h3>Transformer-based LLMs rely on probabilistic decoding, and their stability depends on maintaining a workable entropy margin in the next-token distribution. Recent analyses have shown that when entropy collapses below a minimal threshold, safety masks and pruning mechanisms begin to conflict with the model’s logits. This mechanism explains both short-lived coherence in low-entropy settings and the subsequent onset of deviation, drift, or misinterpretation.</h3> <h3>This technical note extends that work by proposing <strong>discipline-specific entropy bands</strong> for stable scientific reasoning. Different fields place different demands on precision, abstraction, and flexibility. As a result, the entropy range required for coherent reasoning is not uniform across domains.</h3> <h3>-Mathematical reasoning requires a narrow, low-entropy band.</h3> <h3>-General scientific work requires a broader range.</h3> <h3>-Cross-disciplinary conceptual synthesis requires the upper part of the stable region.</h3> <h3>The paper defines three such bands and relates them to the minimal entropy threshold and the Grandić Tunnel Stability Region (GTSR). The analysis clarifies why LLM performance varies across tasks even when prompts are equally clear, and why pushing models into extreme low-entropy regimes can only be sustained briefly under strong external constraints.</h3> <h3><em><strong>This note completes a set of foundational tools for understanding the geometric and thermodynamic boundaries of probabilistic transformer models.</strong></em></h3> |
| title | Entropy Bands for Scientific Reasoning in Probabilistic Transformer Models |
| url | https://doi.org/10.5281/zenodo.17730827 |