Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.14604 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913683631767552 |
|---|---|
| author | Mao, Minjia Wei, Dongjun Chen, Zeyu Fang, Xiao Chau, Michael |
| author_facet | Mao, Minjia Wei, Dongjun Chen, Zeyu Fang, Xiao Chau, Michael |
| contents | Recent advancements in large language models (LLMs) have highlighted the risk of misusing them, raising the need for accurate detection of LLM-generated content. In response, a viable solution is to inject imperceptible identifiers into LLMs, known as watermarks. Our research extends the existing watermarking methods by proposing the novel Sampling One Then Accepting (STA-1) method. STA-1 is an unbiased watermark that preserves the original token distribution in expectation and has a lower risk of producing unsatisfactory outputs in low-entropy scenarios compared to existing unbiased watermarks. In watermark detection, STA-1 does not require prompts or a white-box LLM, provides statistical guarantees, demonstrates high efficiency in detection time, and remains robust against various watermarking attacks. Experimental results on low-entropy and high-entropy datasets demonstrate that STA-1 achieves the above properties simultaneously, making it a desirable solution for watermarking LLMs. Implementation codes for this study are available online. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14604 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Watermarking Low-entropy Generation for Large Language Models: An Unbiased and Low-risk Method Mao, Minjia Wei, Dongjun Chen, Zeyu Fang, Xiao Chau, Michael Computation and Language Recent advancements in large language models (LLMs) have highlighted the risk of misusing them, raising the need for accurate detection of LLM-generated content. In response, a viable solution is to inject imperceptible identifiers into LLMs, known as watermarks. Our research extends the existing watermarking methods by proposing the novel Sampling One Then Accepting (STA-1) method. STA-1 is an unbiased watermark that preserves the original token distribution in expectation and has a lower risk of producing unsatisfactory outputs in low-entropy scenarios compared to existing unbiased watermarks. In watermark detection, STA-1 does not require prompts or a white-box LLM, provides statistical guarantees, demonstrates high efficiency in detection time, and remains robust against various watermarking attacks. Experimental results on low-entropy and high-entropy datasets demonstrate that STA-1 achieves the above properties simultaneously, making it a desirable solution for watermarking LLMs. Implementation codes for this study are available online. |
| title | Watermarking Low-entropy Generation for Large Language Models: An Unbiased and Low-risk Method |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.14604 |