Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.12728 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911102594449408 |
|---|---|
| author | Sivan, Doron Tsodyks, Misha |
| author_facet | Sivan, Doron Tsodyks, Misha |
| contents | In Shannon's seminal paper, entropy of printed English, treated as a stationary stochastic process, was estimated to be roughly 1 bit per character. However, considered as a means of communication, language differs considerably from its printed form: (i) the units of information are not characters or even words but clauses, i.e. shortest meaningful parts of speech; and (ii) what is transmitted is principally the meaning of what is being said or written, while the precise phrasing that was used to communicate the meaning is typically ignored. In this study, we show that one can leverage recently developed large language models to quantify information communicated in meaningful narratives in terms of bits of meaning per clause. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_12728 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Information Theory of Meaningful Communication Sivan, Doron Tsodyks, Misha Computation and Language Information Theory In Shannon's seminal paper, entropy of printed English, treated as a stationary stochastic process, was estimated to be roughly 1 bit per character. However, considered as a means of communication, language differs considerably from its printed form: (i) the units of information are not characters or even words but clauses, i.e. shortest meaningful parts of speech; and (ii) what is transmitted is principally the meaning of what is being said or written, while the precise phrasing that was used to communicate the meaning is typically ignored. In this study, we show that one can leverage recently developed large language models to quantify information communicated in meaningful narratives in terms of bits of meaning per clause. |
| title | Information Theory of Meaningful Communication |
| topic | Computation and Language Information Theory |
| url | https://arxiv.org/abs/2411.12728 |