Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2311.09832 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914826217848832 |
|---|---|
| author | Chen, Liang Bian, Yatao Deng, Yang Cai, Deng Li, Shuaiyi Zhao, Peilin Wong, Kam-fai |
| author_facet | Chen, Liang Bian, Yatao Deng, Yang Cai, Deng Li, Shuaiyi Zhao, Peilin Wong, Kam-fai |
| contents | Text watermarking has emerged as a pivotal technique for identifying machine-generated text. However, existing methods often rely on arbitrary vocabulary partitioning during decoding to embed watermarks, which compromises the availability of suitable tokens and significantly degrades the quality of responses. This study assesses the impact of watermarking on different capabilities of large language models (LLMs) from a cognitive science lens. Our finding highlights a significant disparity; knowledge recall and logical reasoning are more adversely affected than language generation. These results suggest a more profound effect of watermarking on LLMs than previously understood. To address these challenges, we introduce Watermarking with Mutual Exclusion (WatME), a novel approach leveraging linguistic prior knowledge of inherent lexical redundancy in LLM vocabularies to seamlessly integrate watermarks. Specifically, WatME dynamically optimizes token usage during the decoding process by applying a mutually exclusive rule to the identified lexical redundancies. This strategy effectively prevents the unavailability of appropriate tokens and preserves the expressive power of LLMs. We provide both theoretical analysis and empirical evidence showing that WatME effectively preserves the diverse capabilities of LLMs while ensuring watermark detectability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09832 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | WatME: Towards Lossless Watermarking Through Lexical Redundancy Chen, Liang Bian, Yatao Deng, Yang Cai, Deng Li, Shuaiyi Zhao, Peilin Wong, Kam-fai Computation and Language Text watermarking has emerged as a pivotal technique for identifying machine-generated text. However, existing methods often rely on arbitrary vocabulary partitioning during decoding to embed watermarks, which compromises the availability of suitable tokens and significantly degrades the quality of responses. This study assesses the impact of watermarking on different capabilities of large language models (LLMs) from a cognitive science lens. Our finding highlights a significant disparity; knowledge recall and logical reasoning are more adversely affected than language generation. These results suggest a more profound effect of watermarking on LLMs than previously understood. To address these challenges, we introduce Watermarking with Mutual Exclusion (WatME), a novel approach leveraging linguistic prior knowledge of inherent lexical redundancy in LLM vocabularies to seamlessly integrate watermarks. Specifically, WatME dynamically optimizes token usage during the decoding process by applying a mutually exclusive rule to the identified lexical redundancies. This strategy effectively prevents the unavailability of appropriate tokens and preserves the expressive power of LLMs. We provide both theoretical analysis and empirical evidence showing that WatME effectively preserves the diverse capabilities of LLMs while ensuring watermark detectability. |
| title | WatME: Towards Lossless Watermarking Through Lexical Redundancy |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2311.09832 |