Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.04272 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917795793469440 |
|---|---|
| author | Ward, Francis Rhys Yang, Zejia Jackson, Alex Brown, Randy Smith, Chandler Colverd, Grace Thomson, Louis Douglas, Raymond Bartak, Patrik Rowan, Andrew |
| author_facet | Ward, Francis Rhys Yang, Zejia Jackson, Alex Brown, Randy Smith, Chandler Colverd, Grace Thomson, Louis Douglas, Raymond Bartak, Patrik Rowan, Andrew |
| contents | Language models (LMs) can exhibit human-like behaviour, but it is unclear how to describe this behaviour without undue anthropomorphism. We formalise a behaviourist view of LM character traits: qualities such as truthfulness, sycophancy, or coherent beliefs and intentions, which may manifest as consistent patterns of behaviour. Our theory is grounded in empirical demonstrations of LMs exhibiting different character traits, such as accurate and logically coherent beliefs, and helpful and harmless intentions. We find that the consistency with which LMs exhibit certain character traits varies with model size, fine-tuning, and prompting. In addition to characterising LM character traits, we evaluate how these traits develop over the course of an interaction. We find that traits such as truthfulness and harmfulness can be stationary, i.e., consistent over an interaction, in certain contexts, but may be reflective in different contexts, meaning they mirror the LM's behavior in the preceding interaction. Our formalism enables us to describe LM behaviour precisely in intuitive language, without undue anthropomorphism. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04272 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating Language Model Character Traits Ward, Francis Rhys Yang, Zejia Jackson, Alex Brown, Randy Smith, Chandler Colverd, Grace Thomson, Louis Douglas, Raymond Bartak, Patrik Rowan, Andrew Computation and Language Language models (LMs) can exhibit human-like behaviour, but it is unclear how to describe this behaviour without undue anthropomorphism. We formalise a behaviourist view of LM character traits: qualities such as truthfulness, sycophancy, or coherent beliefs and intentions, which may manifest as consistent patterns of behaviour. Our theory is grounded in empirical demonstrations of LMs exhibiting different character traits, such as accurate and logically coherent beliefs, and helpful and harmless intentions. We find that the consistency with which LMs exhibit certain character traits varies with model size, fine-tuning, and prompting. In addition to characterising LM character traits, we evaluate how these traits develop over the course of an interaction. We find that traits such as truthfulness and harmfulness can be stationary, i.e., consistent over an interaction, in certain contexts, but may be reflective in different contexts, meaning they mirror the LM's behavior in the preceding interaction. Our formalism enables us to describe LM behaviour precisely in intuitive language, without undue anthropomorphism. |
| title | Evaluating Language Model Character Traits |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.04272 |