Isotropy Matters: Soft-ZCA Whitening of Embeddings for Semantic Code Search
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913588222885888 |
|---|---|
| author | Diera, Andor Galke, Lukas Scherp, Ansgar |
| author_facet | Diera, Andor Galke, Lukas Scherp, Ansgar |
| contents | Low isotropy in an embedding space impairs performance on tasks involving semantic inference. Our study investigates the impact of isotropy on semantic code search performance and explores post-processing techniques to mitigate this issue. We analyze various code language models, examine isotropy in their embedding spaces, and its influence on search effectiveness. We propose a modified ZCA whitening technique to control isotropy levels in embeddings. Our results demonstrate that Soft-ZCA whitening improves the performance of pre-trained code language models and can complement contrastive fine-tuning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17538 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Isotropy Matters: Soft-ZCA Whitening of Embeddings for Semantic Code Search Diera, Andor Galke, Lukas Scherp, Ansgar Computation and Language Low isotropy in an embedding space impairs performance on tasks involving semantic inference. Our study investigates the impact of isotropy on semantic code search performance and explores post-processing techniques to mitigate this issue. We analyze various code language models, examine isotropy in their embedding spaces, and its influence on search effectiveness. We propose a modified ZCA whitening technique to control isotropy levels in embeddings. Our results demonstrate that Soft-ZCA whitening improves the performance of pre-trained code language models and can complement contrastive fine-tuning. |
| title | Isotropy Matters: Soft-ZCA Whitening of Embeddings for Semantic Code Search |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.17538 |