Evaluating Steering Techniques using Human Similarity Judgments
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908379092353024 |
|---|---|
| author | Studdiford, Zach Rogers, Timothy T. Suresh, Siddharth Mukherjee, Kushin |
| author_facet | Studdiford, Zach Rogers, Timothy T. Suresh, Siddharth Mukherjee, Kushin |
| contents | Current evaluations of Large Language Model (LLM) steering techniques focus on task-specific performance, overlooking how well steered representations align with human cognition. Using a well-established triadic similarity judgment task, we assessed steered LLMs on their ability to flexibly judge similarity between concepts based on size or kind. We found that prompt-based steering methods outperformed other methods both in terms of steering accuracy and model-to-human alignment. We also found LLMs were biased towards 'kind' similarity and struggled with 'size' alignment. This evaluation approach, grounded in human cognition, adds further support to the efficacy of prompt-based steering and reveals privileged representational axes in LLMs prior to steering. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19333 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Evaluating Steering Techniques using Human Similarity Judgments Studdiford, Zach Rogers, Timothy T. Suresh, Siddharth Mukherjee, Kushin Artificial Intelligence I.2.7 Current evaluations of Large Language Model (LLM) steering techniques focus on task-specific performance, overlooking how well steered representations align with human cognition. Using a well-established triadic similarity judgment task, we assessed steered LLMs on their ability to flexibly judge similarity between concepts based on size or kind. We found that prompt-based steering methods outperformed other methods both in terms of steering accuracy and model-to-human alignment. We also found LLMs were biased towards 'kind' similarity and struggled with 'size' alignment. This evaluation approach, grounded in human cognition, adds further support to the efficacy of prompt-based steering and reveals privileged representational axes in LLMs prior to steering. |
| title | Evaluating Steering Techniques using Human Similarity Judgments |
| topic | Artificial Intelligence I.2.7 |
| url | https://arxiv.org/abs/2505.19333 |