Select or Project? Evaluating Lower-dimensional Vectors for LLM Training Data Explanations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908795472445440 |
|---|---|
| author | Hinterleitner, Lukas Schoenegger, Loris Roth, Benjamin |
| author_facet | Hinterleitner, Lukas Schoenegger, Loris Roth, Benjamin |
| contents | Gradient-based methods for instance-based explanation for large language models (LLMs) are hindered by the immense dimensionality of model gradients. In practice, influence estimation is restricted to a subset of model parameters to make computation tractable, but this subset is often chosen ad hoc and rarely justified by systematic evaluation. This paper investigates if it is better to create low-dimensional representations by selecting a small, architecturally informed subset of model components or by projecting the full gradients into a lower-dimensional space. Using a novel benchmark, we show that a greedily selected subset of components captures the information about training data influence needed for a retrieval task more effectively than either the full gradient or random projection. We further find that this approach is more computationally efficient than random projection, demonstrating that targeted component selection is a practical strategy for making instance-based explanations of large models more computationally feasible. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_16651 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Select or Project? Evaluating Lower-dimensional Vectors for LLM Training Data Explanations Hinterleitner, Lukas Schoenegger, Loris Roth, Benjamin Computation and Language I.2.7 Gradient-based methods for instance-based explanation for large language models (LLMs) are hindered by the immense dimensionality of model gradients. In practice, influence estimation is restricted to a subset of model parameters to make computation tractable, but this subset is often chosen ad hoc and rarely justified by systematic evaluation. This paper investigates if it is better to create low-dimensional representations by selecting a small, architecturally informed subset of model components or by projecting the full gradients into a lower-dimensional space. Using a novel benchmark, we show that a greedily selected subset of components captures the information about training data influence needed for a retrieval task more effectively than either the full gradient or random projection. We further find that this approach is more computationally efficient than random projection, demonstrating that targeted component selection is a practical strategy for making instance-based explanations of large models more computationally feasible. |
| title | Select or Project? Evaluating Lower-dimensional Vectors for LLM Training Data Explanations |
| topic | Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2601.16651 |