Statistical Foundations of DIME: Risk Estimation for Practical Index Selection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918437467455488 |
|---|---|
| author | D'Erasmo, Giulio Campagnano, Cesare Mallia, Antonio Brutti, Pierpaolo Tonellotto, Nicola Silvestri, Fabrizio |
| author_facet | D'Erasmo, Giulio Campagnano, Cesare Mallia, Antonio Brutti, Pierpaolo Tonellotto, Nicola Silvestri, Fabrizio |
| contents | High-dimensional dense embeddings have become central to modern Information Retrieval, but many dimensions are noisy or redundant. Recently proposed DIME (Dimension IMportance Estimation), provides query-dependent scores to identify informative components of embeddings. DIME relies on a costly grid search to select a priori a dimensionality for all the query corpus's embeddings. Our work provides a statistically grounded criterion that directly identifies the optimal set of dimensions for each query at inference time. Experiments confirm achieving parity of effectiveness and reduces embedding size by an average of $\sim50\%$ across different models and datasets at inference time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_05649 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Statistical Foundations of DIME: Risk Estimation for Practical Index Selection D'Erasmo, Giulio Campagnano, Cesare Mallia, Antonio Brutti, Pierpaolo Tonellotto, Nicola Silvestri, Fabrizio Information Retrieval High-dimensional dense embeddings have become central to modern Information Retrieval, but many dimensions are noisy or redundant. Recently proposed DIME (Dimension IMportance Estimation), provides query-dependent scores to identify informative components of embeddings. DIME relies on a costly grid search to select a priori a dimensionality for all the query corpus's embeddings. Our work provides a statistically grounded criterion that directly identifies the optimal set of dimensions for each query at inference time. Experiments confirm achieving parity of effectiveness and reduces embedding size by an average of $\sim50\%$ across different models and datasets at inference time. |
| title | Statistical Foundations of DIME: Risk Estimation for Practical Index Selection |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2601.05649 |