Corrections of Zipf's and Heaps' Laws Derived from Hapax Rate Models
Fuente:
arXiv
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908376758222848 |
|---|---|
| author | Dębowski, Łukasz |
| author_facet | Dębowski, Łukasz |
| contents | The article introduces corrections to Zipf's and Heaps' laws based on systematic models of the proportion of hapaxes, i.e., words that occur once. The derivation rests on two assumptions: The first one is the standard urn model which predicts that marginal frequency distributions for shorter texts look as if word tokens were sampled blindly from a given longer text. The second assumption posits that the hapax rate is a simple function of the text length. Four such functions are discussed: the constant model, the Davis model, the linear model, and the logistic model. It is shown that the logistic model yields the best fit. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_12896 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Corrections of Zipf's and Heaps' Laws Derived from Hapax Rate Models Dębowski, Łukasz Computation and Language Applications 62P99 I.2.7 The article introduces corrections to Zipf's and Heaps' laws based on systematic models of the proportion of hapaxes, i.e., words that occur once. The derivation rests on two assumptions: The first one is the standard urn model which predicts that marginal frequency distributions for shorter texts look as if word tokens were sampled blindly from a given longer text. The second assumption posits that the hapax rate is a simple function of the text length. Four such functions are discussed: the constant model, the Davis model, the linear model, and the logistic model. It is shown that the logistic model yields the best fit. |
| title | Corrections of Zipf's and Heaps' Laws Derived from Hapax Rate Models |
| topic | Computation and Language Applications 62P99 I.2.7 |
| url | https://arxiv.org/abs/2307.12896 |