Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916683915984896 |
|---|---|
| author | Linhardt, Lorenz Neuhäuser, Tom Tětková, Lenka Eberle, Oliver |
| author_facet | Linhardt, Lorenz Neuhäuser, Tom Tětková, Lenka Eberle, Oliver |
| contents | Small and mid-sized generative language models have gained increasing attention. Their size and availability make them amenable to being analyzed at a behavioral as well as a representational level, allowing investigations of how these levels interact. We evaluate 32 publicly available language models for their representational and behavioral alignment with human similarity judgments on a word triplet task. This provides a novel evaluation setting to probe semantic associations in language beyond common pairwise comparisons. We find that (1) even the representations of small language models can achieve human-level alignment, (2) instruction-tuned model variants can exhibit substantially increased agreement, (3) the pattern of alignment across layers is highly model dependent, and (4) alignment based on models' behavioral responses is highly dependent on model size, matching their representational alignment only for the largest evaluated models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07965 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments Linhardt, Lorenz Neuhäuser, Tom Tětková, Lenka Eberle, Oliver Machine Learning Computation and Language Small and mid-sized generative language models have gained increasing attention. Their size and availability make them amenable to being analyzed at a behavioral as well as a representational level, allowing investigations of how these levels interact. We evaluate 32 publicly available language models for their representational and behavioral alignment with human similarity judgments on a word triplet task. This provides a novel evaluation setting to probe semantic associations in language beyond common pairwise comparisons. We find that (1) even the representations of small language models can achieve human-level alignment, (2) instruction-tuned model variants can exhibit substantially increased agreement, (3) the pattern of alignment across layers is highly model dependent, and (4) alignment based on models' behavioral responses is highly dependent on model size, matching their representational alignment only for the largest evaluated models. |
| title | Cat, Rat, Meow: On the Alignment of Language Model and Human Term-Similarity Judgments |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2504.07965 |