C2T-ID: Converting Semantic Codebooks to Textual Document Identifiers for Generative Search
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911226322223104 |
|---|---|
| author | Zhang, Yingchen Zhang, Ruqing Guo, Jiafeng Peng, Wenjun Li, Sen Lv, Fuyu Cheng, Xueqi |
| author_facet | Zhang, Yingchen Zhang, Ruqing Guo, Jiafeng Peng, Wenjun Li, Sen Lv, Fuyu Cheng, Xueqi |
| contents | Designing document identifiers (docids) that carry rich semantic information while maintaining tractable search spaces is a important challenge in generative retrieval (GR). Popular codebook methods address this by building a hierarchical semantic tree and constraining generation to its child nodes, yet their numeric identifiers cannot leverage the large language model's pretrained natural language understanding. Conversely, using text as docid provides more semantic expressivity but inflates the decoding space, making the system brittle to early-step errors. To resolve this trade-off, we propose C2T-ID: (i) first construct semantic numerical docid via hierarchical clustering; (ii) then extract high-frequency metadata keywords and iteratively replace each numeric label with its cluster's top-K keywords; and (iii) an optional two-level semantic smoothing step further enhances the fluency of C2T-ID. Experiments on Natural Questions and Taobao's product search demonstrate that C2T-ID significantly outperforms atomic, semantic codebook, and pure-text docid baselines, demonstrating its effectiveness in balancing semantic expressiveness with search space constraints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_19221 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | C2T-ID: Converting Semantic Codebooks to Textual Document Identifiers for Generative Search Zhang, Yingchen Zhang, Ruqing Guo, Jiafeng Peng, Wenjun Li, Sen Lv, Fuyu Cheng, Xueqi Information Retrieval Designing document identifiers (docids) that carry rich semantic information while maintaining tractable search spaces is a important challenge in generative retrieval (GR). Popular codebook methods address this by building a hierarchical semantic tree and constraining generation to its child nodes, yet their numeric identifiers cannot leverage the large language model's pretrained natural language understanding. Conversely, using text as docid provides more semantic expressivity but inflates the decoding space, making the system brittle to early-step errors. To resolve this trade-off, we propose C2T-ID: (i) first construct semantic numerical docid via hierarchical clustering; (ii) then extract high-frequency metadata keywords and iteratively replace each numeric label with its cluster's top-K keywords; and (iii) an optional two-level semantic smoothing step further enhances the fluency of C2T-ID. Experiments on Natural Questions and Taobao's product search demonstrate that C2T-ID significantly outperforms atomic, semantic codebook, and pure-text docid baselines, demonstrating its effectiveness in balancing semantic expressiveness with search space constraints. |
| title | C2T-ID: Converting Semantic Codebooks to Textual Document Identifiers for Generative Search |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2510.19221 |