Distributional Information Embedding: A Framework for Multi-bit Watermarking
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918078706614272 |
|---|---|
| author | He, Haiyun Liu, Yepeng Wang, Ziqiao Mao, Yongyi Bu, Yuheng |
| author_facet | He, Haiyun Liu, Yepeng Wang, Ziqiao Mao, Yongyi Bu, Yuheng |
| contents | This paper introduces a novel problem, distributional information embedding, motivated by the practical demands of multi-bit watermarking for large language models (LLMs). Unlike traditional information embedding, which embeds information into a pre-existing host signal, LLM watermarking actively controls the text generation process--adjusting the token distribution--to embed a detectable signal. We develop an information-theoretic framework to analyze this distributional information embedding problem, characterizing the fundamental trade-offs among three critical performance metrics: text quality, detectability, and information rate. In the asymptotic regime, we demonstrate that the maximum achievable rate with vanishing error corresponds to the entropy of the LLM's output distribution and increases with higher allowable distortion. We also characterize the optimal watermarking scheme to achieve this rate. Extending the analysis to the finite-token case with non-i.i.d. tokens, we identify schemes that maximize detection probability while adhering to constraints on false alarm and distortion. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_16558 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Distributional Information Embedding: A Framework for Multi-bit Watermarking He, Haiyun Liu, Yepeng Wang, Ziqiao Mao, Yongyi Bu, Yuheng Cryptography and Security Information Theory Machine Learning This paper introduces a novel problem, distributional information embedding, motivated by the practical demands of multi-bit watermarking for large language models (LLMs). Unlike traditional information embedding, which embeds information into a pre-existing host signal, LLM watermarking actively controls the text generation process--adjusting the token distribution--to embed a detectable signal. We develop an information-theoretic framework to analyze this distributional information embedding problem, characterizing the fundamental trade-offs among three critical performance metrics: text quality, detectability, and information rate. In the asymptotic regime, we demonstrate that the maximum achievable rate with vanishing error corresponds to the entropy of the LLM's output distribution and increases with higher allowable distortion. We also characterize the optimal watermarking scheme to achieve this rate. Extending the analysis to the finite-token case with non-i.i.d. tokens, we identify schemes that maximize detection probability while adhering to constraints on false alarm and distortion. |
| title | Distributional Information Embedding: A Framework for Multi-bit Watermarking |
| topic | Cryptography and Security Information Theory Machine Learning |
| url | https://arxiv.org/abs/2501.16558 |