Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.14868 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911275241439232 |
|---|---|
| author | Ding, Xueying Huang, Xingyue Ju, Mingxuan Collins, Liam Liu, Yozen Akoglu, Leman Shah, Neil Zhao, Tong |
| author_facet | Ding, Xueying Huang, Xingyue Ju, Mingxuan Collins, Liam Liu, Yozen Akoglu, Leman Shah, Neil Zhao, Tong |
| contents | Large language models produce powerful text embeddings, but their causal attention mechanism restricts the flow of information from later to earlier tokens, degrading representation quality. While recent methods attempt to solve this by prepending a single summary token, they over-compress information, hence harming performance on long documents. We propose Hierarchical Token Prepending (HTP), a method that resolves two critical bottlenecks. To mitigate attention-level compression, HTP partitions the input into blocks and prepends block-level summary tokens to subsequent blocks, creating multiple pathways for backward information flow. To address readout-level over-squashing, we replace last-token pooling with mean-pooling, a choice supported by theoretical analysis. HTP achieves consistent performance gains across 11 retrieval datasets and 30 general embedding benchmarks, especially in long-context settings. As a simple, architecture-agnostic method, HTP enhances both zero-shot and finetuned models, offering a scalable route to superior long-document embeddings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_14868 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings Ding, Xueying Huang, Xingyue Ju, Mingxuan Collins, Liam Liu, Yozen Akoglu, Leman Shah, Neil Zhao, Tong Computation and Language Machine Learning Large language models produce powerful text embeddings, but their causal attention mechanism restricts the flow of information from later to earlier tokens, degrading representation quality. While recent methods attempt to solve this by prepending a single summary token, they over-compress information, hence harming performance on long documents. We propose Hierarchical Token Prepending (HTP), a method that resolves two critical bottlenecks. To mitigate attention-level compression, HTP partitions the input into blocks and prepends block-level summary tokens to subsequent blocks, creating multiple pathways for backward information flow. To address readout-level over-squashing, we replace last-token pooling with mean-pooling, a choice supported by theoretical analysis. HTP achieves consistent performance gains across 11 retrieval datasets and 30 general embedding benchmarks, especially in long-context settings. As a simple, architecture-agnostic method, HTP enhances both zero-shot and finetuned models, offering a scalable route to superior long-document embeddings. |
| title | Hierarchical Token Prepending: Enhancing Information Flow in Decoder-based LLM Embeddings |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2511.14868 |