Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.11334 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914604017254400 |
|---|---|
| author | Pereg, Deborah Wand, Michael |
| author_facet | Pereg, Deborah Wand, Michael |
| contents | An information-theoretic framework is introduced to analyze last-layer embedding, focusing on learned representations for regression tasks. We define representation-rate and derive limits on the reliability with which input-output information can be represented as is inherently determined by the input-source entropy. We further define representation capacity in a perturbed setting, and representation rate-distortion for a compressed output. We derive the achievable capacity, the achievable representation-rate, and their converse. Finally, we combine the results in a unified setting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_11334 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Information Theoretic Perspective on Representation Learning Pereg, Deborah Wand, Michael Information Theory Machine Learning An information-theoretic framework is introduced to analyze last-layer embedding, focusing on learned representations for regression tasks. We define representation-rate and derive limits on the reliability with which input-output information can be represented as is inherently determined by the input-source entropy. We further define representation capacity in a perturbed setting, and representation rate-distortion for a compressed output. We derive the achievable capacity, the achievable representation-rate, and their converse. Finally, we combine the results in a unified setting. |
| title | Information Theoretic Perspective on Representation Learning |
| topic | Information Theory Machine Learning |
| url | https://arxiv.org/abs/2601.11334 |